"Why is our datacenter bill so insane?" "We confused DR with DNR."



  • I think that's about what I pay. When I got quotes for anything over 400cc it was more like $700+ a year. I'm not sure why since I have no accidents and no tickets, but maybe my insurance went up after my last bike was stolen.



  • @smallshellscript said:

    This is one of those things you have to have to check a box on the CIO's list of toys all my friends have so I want one too or I'm gonna throw the mother of all tantrums in the toy store. But no one actually tests it until you need it and that's when you find out that the backups are being piped to dev/nul and the CIO's intern nephew has been using the space to host the biggest warez site on the 'net.

    Unless you work where I do. You see, we actually bother to test our DR...

    @Intercourse said:

    It is a shame that the 2-stroke engine went the way of the dodo. I know that they pollute more, etc. Still, there is nothing like the fury of a big 2-stroke engine coming in to its power band.

    EMD 567, 'nuff said. Biggest 2-stroke you'll ever meet.

    @Jaime said:

    Sure, but it's not the same as Weng's situation. UPS doesn't just say "let's have someone else deliver our packages". They deliver most of their packages and use other providers for servicing unusually large loads (like holidays), areas that they can't service well, or to provide services that UPS doesn't do at the moment. The Post Office allows UPS to get into the small package business at around the five dollar price that UPS just doesn't have the systems in place to do.

    They also outsource because they don't have the capacity to do some things nearly as efficiently in-house; rail intermodal is several times more efficient than cross-country trucks for long-haul work.



  • @tarunik said:

    Unless you work where I do. You see, we actually bother to test our DR...

    What is this "test" you speak of?

    I can just see what would happen in the event of a system destroying event here.

    • first we hope that last week's off-site backup didn't fail because it was bigger than the tape it was trying to fit in.
    • then we wander around trying to find license data and install media for all of the decade+ old operating systems and other sw on most of our "business critical" servers.
    • then we find some kind of incompatibility between the version of Oracle that we've installed on the recovered machines and the one that used to be there so our DBs don't come back cleanly.
    • some other critical system (like the customer database) turns out was never included in the backups at all.

    A month later we're limping along at some semblance of normal.

    This, of course, assumes a software issue that wipes our network (like that one manager who opened the "pics of Anna Kournikova" (no Discourse, not Kalashnikov) that the guy he knew from the ISO committee sent helped propagate the virus, getting our e-mail server blacklisted in the process) - I'm assuming that these things use pictures of Jennifer Lawrence now. If we lose all the hardware, it's 2 months of eBay just to get hardware that's old enough to run this crap and $deity help me when it comes to getting the HP-UX servers back.



  • @smallshellscript said:

    then we find some kind of incompatibility between the version of Oracle that we've installed on the recovered machines and the one that used to be there so our DBs don't come back cleanly.

    I just spent some time today asking for a redesign of a feature for a reason similar to this. Another developer designed an application feature so that it required a piece of third party software to be installed and configured in a specific way for the feature to work. I pointed out that it's pretty easy to implement the feature with no external dependencies if we just change the implementation specifics a little bit. His response: "It only takes a few minutes to set this up". He, of course, forgot that our application takes a half a day to set up and no one ever gets the 76 thousand disconnected steps right. It got that way one "It only takes a few minutes to set this up" at a time.

    Design stuff to be easy to DR and you'll never be sorry. Even better, use your DR strategy as a deployment strategy - deploy to the other instance and flop over to go live. Never do an in-place upgrade of the underlying technology - install a fresh platform and deploy to it.



  • @Cursorkeys said:

    Power band is a misnomer on those bikes, sounds too nice, it should be called a power discontinuity.

    The Millennium Falcon was clearly a two-stroke.


  • ♿ (Parody)

    @Jaime said:

    Even better, use your DR strategy as a deployment strategy - deploy to the other instance and flop over to go live.

    We don't quite do that, but we use our DR environment basically as a UAC environment / sandbox for them to try stuff and for training staff. So at least we know it works.



  • @smallshellscript said:

    What is this "test" you speak of?

    We fail a subset of our production systems over to DR oh...quarterly or so? And it's a different subset each quarter, so by the time the year ends, everything that's important has been failover-tested.

    Of course, that's not counting operational DR, which is a thing for us as well...

    @smallshellscript said:

    If we lose all the hardware, it's 2 months of eBay just to get hardware that's old enough to run this crap and $deity help me when it comes to getting the HP-UX servers back.

    Heck, we've been buying up VAX spares for some of our systems...

    @Jaime said:

    Design stuff to be easy to DR and you'll never be sorry. Even better, use your DR strategy as a deployment strategy - deploy to the other instance and flop over to go live. Never do an in-place upgrade of the underlying technology - install a fresh platform and deploy to it.

    So much this. Then again, some systems are complicated to DR for reasons you cannot control, like having to switch physical field hardware over to start talking to the DR site...



  • @tarunik said:

    Then again, some systems are complicated to DR for reasons you cannot control, like having to switch physical field hardware over to start talking to the DR site

    That's why you make it part of your routine instead of a I-hope-this-never-happens event. You'll end up putting in proxies or using DNS entries to do a centralized reconfiguration instead of tolerating reconfiguring individual devices.



  • @Jaime said:

    That's like UPS outsourcing package delivery.

    Ok kind of non-sequitur, but I had to put the idea somewhere once it came up. ;-)

    @Jaime said:

    You'll end up putting in proxies or using DNS entries to do a centralized reconfiguration instead of tolerating reconfiguring individual devices.

    This. I really wish our core processor would do this.



  • @Jaime said:

    That's why you make it part of your routine instead of a I-hope-this-never-happens event. You'll end up putting in proxies or using DNS entries to do a centralized reconfiguration instead of tolerating reconfiguring individual devices.

    Follow-up: the field devices in question don't have support for name resolution -- they're configured with a numeric IP address. So, proxies are used for the DR testing -- but they live at the main site.



  • Have a Nice Post for reminding me of wtf 😃

    @accalia said:

    ```text
    accalia@Devbox2:$ wtf is JDGI
    JDGI: Jeff Doesn't Get It
    accalia@Devbox2:$

    who else here is a TDWTFer, cause i didn't add that one to the acronym database on our devbox...</blockquote>
    
    ```text
    $ wtf is JDGI
    Gee...  I don't know what JDGI means...
    

    ENOREPRO 😢


  • FoxDev

    it's not part of the standard WTF database. but someone added it to the one on Devbox2 (but not Devbox1 or Devbox3, i checked.)



  • @henke37 said:

    What's "DR"?

    Damage Reduction (e.g x/-).



  • Wasn't me, I don't work at your place, but someone else clearly does... The plot thickens, and I love the intrigue!



  • Given the consequences of this for the business, I'm guessing 'DNR' means 'Do Not Resuscitate' here.


  • Garbage Person

    Fuck is I remember. I think someone edited the title. The joke is way lamer than I usually stoop to.


  • FoxDev

    @ScholRLEA said:

    Given the consequences of this for the business, I'm guessing 'DNR' means 'Do Not Resuscitate' here.

    That is the usual expansion of the acronym in every context i am aware of.


  • Garbage Person

    Department of Natural Resources


  • FoxDev

    hmm....

    i'll grant you that one....

    not that it's exactly common to hear (IIRC our local one is Department of Environment or some such, which causes no end of confusiuon with the Department of Education and the department of Energy when acronymized)



  • On a remotely related note. One of my friends had worked in a company as IT support. Their team did excellent job that all of their production systems have 100% uptime over 3 years, despite of on-going server/network infrastructure upgrades thanks to good planning, and the fault tolarance technologies.

    Then their top-paid support staffs (half of their team, if I remember it correctly) were layoffed because the HR say they have evidance that they don't need that much man power.

    That's why we can't have nice things.


Log in to reply