"Why is our datacenter bill so insane?" "We confused DR with DNR."


  • Garbage Person

    Our primary datacenter is all-outsourced. Production disk is expensive as hell - think middle six figures for 5TB for a year. VM's are also quite pricey, but not nearly that distorted.

    And then I noticed:
    There are only two storage tiers. Prod and non-prod. The difference being that Prod is live-synchronized to the DR site. In realtime.

    We're putting operating system disks on that.
    We're putting application servers with no moving data on that.
    We're putting logs on that.
    We're putting ever-accumulating archival cruft on that.

    So I was talking about adding many, many new application VM's. Upgrades for the software in question come once every two or three years (and are usually applied by 'install new machines with the new version because recertifying everything for the new version would be too costly). So I suggested we simply mirror the VM every time an OS patch goes in - once a month.

    "That adds too much complexity. Complexity costs money. And it isn't in line with our business continuity plans."

    DID YOU KNOW! Realtime synchronization of the pagefile (but not the active RAM, so we can't seamlessly failover) is essential for business continuity.



  • "Does that complexity cost more money than we're wasting right now?"



  • Shhh. If I reduce costs without being asked, they'll cut my budget and then my manager-peen will be smaller than the next guy's.

    Edit: apostrophe



  • I know about expensive disks, but even for 100% SSDs, DR'd and backed up, USD 100 per GB per year is kinda outside normal limits. . . Unless maybe you only have 5 TB and 50 VMs and you're bearing the whole brunt of a 24/24 team to look after them? Or something . . .


  • Garbage Person

    I asked that, too. The excuse was some blather about human error and procedures will not be followed.

    My final words on the matter were something about how this is a solved, automated problem and even if it weren't that humans who fail to execute procedure should see some sort of consequence.

    This 'nobody is held accountable for even the most catastrophic and unforgivable fuckups' thing is a theme that I'm running with this year, because nobody is held accountable and it's really starting to piss me off. I know of people who make $18/hr who have demonstrably (as in there are logs of them clicking the button to pass through the dire warnings about the repercussions of the stupid thing they're doing) cost the company millions in material writeoffs, overtime and penalties and then denied ever having done it (must have been someone using their account!) who don't have a single black mark whatsoever. Or even a 'had to retake password security training'.

    We have some 4000 VM's in this datacenter. I am {mumble}responsible{/mumble} for like eighty of them. Because of cost, getting a few terabytes added for short term demand is a CIO-level decision.

    Also, the backups that are included in that price are host-based backups. And they are notorious for just not fucking working. I've had several servers recently where they're all "WE NEED TO OUT OF CYCLE REBOOT YOUR SERVER TO UPDATE VMTOOLS TO START DIAGNOSIS ON YOUR BACKUP!"

    Including my primary fileserver. Which went un-backed-up for a year (I noticed because I tried to restore a file and "it ain't there" was the answer. Nobody took this as a serious event until after I had basically called God about it.) And it's godawful symantec backup shit, so it consumes ungodly amounts of RAM and CPU calculating diffs on high-churn disks (which my fileserver very much is). It just about finishes it's daily differential in 24 hours.

    Our support team vendor side seems to have about fifteen people on it. Five DBA's, five hardware wranglers/reboot-button-pushers, and five managers/paper pushers. Maybe an intern who loads the tapes onto the Iron Mountain truck.



  • @smallshellscript said:

    Shhh. If I reduce costs without being asked, they'll cut my budget and then my manager-peen will be smaller than the next guys.

    "Cutting costs could also mean you will have budget to hire more people and/or address other problems. Over a few years, these savings could equal two Ferraris*. I'll take the other one if you don't want it."

    @Weng said:

    I asked that, too. The excuse was some blather about human error and procedures will not be followed.

    "Automating the process will both establish procedure (or enforce existing procedure if desired) and minimize human error. There's always risk in cutting new code, but that risk could mean saving big bucks."

    @Weng said:

    This 'nobody is held accountable for even the most catastrophic and unforgivable fuckups' thing is a theme that I'm running with this year, because nobody is held accountable and it's really starting to piss me off.

    Especially for the guy who has to clean up after said fuckups.

    *In no way do I condone corruption or misappropriation of company funds. Especially not on overpriced Italian cars.



  • @Groaner said:

    Cutting costs could also mean you will have budget to hire more people and/or address other problems.

    Infrastructure budget cannot be used to increase headcount. We aren't allowed to classify any more interns as servers.



  • @smallshellscript said:

    Shhh. If I reduce costs without being asked, they'll cut my budget and then my manager-peen will be smaller than the next guys.

    They probably already know you're a little deer.


  • Garbage Person

    @Groaner said:

    "Cutting costs could also mean you will have budget to hire more people and/or address other problems. Over a few years, these savings could equal two Ferraris*. I'll take the other one if you don't want it."
    Screw that. If you give me the tools to manage my servers more sanely, I'll use the savings to buy MORE FUCKING SERVERS. You know, so your boss can stop yelling at you for our constant capacity crises and you can stop fucking my development schedule up by having me divert all attention to keeping the goddamn ship afloat.

    My team of highly paid developers really appreciates it when the flow of work for them to do stops because I'm too busy with conference calls and emergency OMFG meetings about job queues and memory utilization to keep them fed with strategic direction (and they're all too new to make independent decisions about what direction our product needs to move in).

    And while we're at it, can we start planning product features more than 2 days in advance? That'd be really fucking cool!

    Oh, and I really like it when the outsourced DBA's tell me to rewrite the app because it DARES to use more than 512mb of TempDB for thousands of simultaneous jobs. If you could schedule me for more conference calls in that vein, I'd greatly appreciate it!


  • Discourse touched me in a no-no place

    @smallshellscript said:

    We aren't allowed to classify any more interns as servers.

    Stop having them deliver coffee, then. Problem solved.



  • @Keith said:

    They probably already know you're a little deer.

    That's business deer


  • Winner of the 2016 Presidential Election

    @Weng said:

    Oh, and I really like it when the outsourced DBA's tell me to rewrite the app because it DARES to use more than 512mb of TempDB for thousands of simultaneous jobs.

    Did you take snoofle's old job?

    [quote="snoofle]
    It turns out that there are four instances of the application that query
    this service, and the DB simply can't handle 120 parallel queries. Mind
    you this is a fairly huge Oracle RAC system with many GB of ram and 128
    cores. However, the DBAs only have 512MB of scratch space allocated.
    For all users. For the entire DB. When you're querying tables with 3+B
    rows, and need subqueries, etc., that scratch space isn't going to get
    you very far.
    [/quote]


  • Garbage Person

    No. We're MSSQL. And I've previously identified what industry I work in. Probably well enough for a determined individual to figure out the company.


  • Winner of the 2016 Presidential Election

    Ah, I missed that. The common scratch/temp space limit of 512MB for a heavily used database is an interesting coincidence.


  • Garbage Person

    It wasn't really 512mb, it was more along the lines of 50gb. We eventually browbeat them into "JUST ADD MORE, WE'LL FIX IT LATER" and got it up to 500gb. Which is roughly 2x average needs.

    Protip: We ain't gonna fix it.



  • Welcome to the business world. Knowing what you do is the first step. Getting someone else (e.g., the PHBs) to understand a situation is more important.



  • What you need to do is somehow discreetly get one of the bigwigs to whine about how much the IT department is spending on the datacenter, then be the big hero and offer a solution.

    Do you have a favor to call in with any of them?


  • Garbage Person

    I don't have many bigwigs above me. It's literally my boss, an SVP, and then the CIO.

    The CIO was involved in arranging the whole catastrophic deal with this provider and is whole hog into getting everything out of our inhouse DC's and into theirs. Not going to find sympathy there.
    The SVP is in the doghouse with the CIO after presiding over a disasterous development effort.
    My boss is in the SVP's doghouse because he neglected to mention several impending catastrophes and got caught out.

    In other words, I don't have enough middle managers available to play that game. Will reconsider it when my team is bounced to another part of the org chart.



  • @Weng said:

    The difference being that Prod is live-synchronized to the DR site. In realtime

    I work at a cloud provider. Our sales guy just promised a big client that we would do this for their 15TB of storage. For no extra charge. Because we like them.

    This requires exactly as much bandwidth between the datacenters as the servers have I/O bandwidth. Not just bandwidth, but low-latency bandwidth. If you need data consistency, then any latency in the network link will be seen as addition I/O latency at the server. I believe that your provider charges a stupid amount of money for this level of redundancy. The real question is; "Why is this your standard and not an option that is only used when appropriate?".

    Also, who outsources 4000 servers? A company wouldn't be the smallest player in the cloud market if they had only you as a client. That's like UPS outsourcing package delivery.


  • Garbage Person

    @Jaime said:

    If you need data consistency, then any latency in the network link will be seen as addition I/O latency at the server. I believe that your provider charges a stupid amount of money for this level of redundancy.
    Yeah. I am 99.99999999% certain that this isn't happening. For as miserable and shitty as these servers perform, there isn't a 1200 mile I/O loop in them. If anything, they're replicating SAN in the background. Of course, it's sold to the application teams that bear the cost as "realtime".

    Actual documented RTO/RPO is 16 hours/12 hours. And you can only failover the whole fucking center. And the number of hosts is smaller, so performance will blow (though this will be balanced somewhat by the total loss of all development, test and "production tier is too expensive, we can live without it" VM's).

    They've allegedly done tests wherein they brought up all the VM's, but left them disconnected from the network. Of course, I have no evidence of that, because a server that's disconnected from the network is really hard to verify functionality on.

    @Jaime said:

    Also, who outsources 4000 servers? A company wouldn't be the smallest player in the cloud market if they had only you as a client. That's like UPS outsourcing package delivery.
    We do! Because we're SMART! Servers cost money! It's not our core competency! It's not what our customers pay us to do (Shut up you in the back!)

    Best part is that our outsourcing provider has exactly one IT Services client: Us. Their primary line of business is, honestly, fucking terrifying. We may as well be using the NSA as cloud storage.

    Our IT HQ building used to be next to theirs. Ours, naturally, didn't have a DC of any note in it.

    So when data center consolidation came up (until then, all our servers were stashed piecemeal in offices, manufacturing sites, etc.), golf games were played and they got the job.


  • Garbage Person

    Oh, and they can't DR physical servers. At all. So I need to take part of my app off a 24-core, 128gb monstrosity and figure out how to cram it into a 12-core max, 32gb max VM. Or buy a duplicate server and duplicate full-prod licenses to the tune of half a squillion bucks. Even though the thing uses DR'd SAN for storage (including the OS).

    Naturally this wasn't mentioned until the thing had been in prod for two fucking years.


  • Discourse touched me in a no-no place

    Sounds like a total clusterfuck of the kind that won't get any traction for fixing until it blows up to the point of making the CEO look like an idiot in front of the press. Sometimes that sort of cold hard reality what it takes to make management look again at the bad choices they've made.



  • @Jaime said:

    That's like UPS outsourcing package delivery.

    [spoiler]They actually do that, you know... 😉 (albeit for long-haul moves, not the actual delivery aspect that gets the package to your door) [/spoiler]

    @Weng said:

    Best part is that our outsourcing provider has exactly one IT Services client: Us. Their primary line of business is, honestly, fucking terrifying. We may as well be using the NSA as cloud storage.

    Defense contractor as cloud provider, eh?


  • Garbage Person

    Worse.



  • @tarunik said:

    They actually do that, you know... 😉 (albeit for long-haul moves, not the actual delivery aspect that gets the package to your door)

    UPS does outsource door deliveries. Usually to the postal service. I've had shipments where the tracking number is UPS, and the last bit of information I've received on the tracking number is "Delivered to local Post Office for final delivery." Then I get the package two or three days later.



  • @abarker said:

    UPS does outsource door deliveries. Usually to the postal service. I've had shipments where the tracking number is UPS, and the last bit of information I've received on the tracking number is "Delivered to local Post Office for final delivery." Then I get the package two or three days later.

    I wish they would do that around here. Then the postal service would try to deliver it while I wasn't home, and leave it in the post office for me to pick up. The way UPS does it now, they usually try to deliver it while I'm not home, and since I'm not there it goes to the "UPS customer center" located at the far end of nowhere.

    Instead of dealing with that bs I bought a postbox at the local UPS store, which accepts packages from all couriers during work hours.


  • Grade A Premium Asshole

    @abarker said:

    UPS does outsource door deliveries.

    Yep. Happens all the time. A lot of the trucks you see on the road are contract trucks also, run by independent owner-operators. Also, a lot of their parcels carried by air in to remote areas (Alaska, Upper Peninsula, etc) are flown by independent carriers. They outsource a little of everything.



  • @abarker said:

    UPS does outsource door deliveries. Usually to the postal service. I've had shipments where the tracking number is UPS, and the last bit of information I've received on the tracking number is "Delivered to local Post Office for final delivery." Then I get the package two or three days later.

    This is a different service than regular UPS Ground or Air, though (it goes by "UPS Mail Innovations").

    @Intercourse said:

    A lot of the trucks you see on the road are contract trucks also, run by independent owner-operators.

    Not terribly surprising, come to think of it; also, they are a major rail intermodal customer (which is what I was originally alluding to), so it's not all that rare to see a bunch of UPS Ground trailers hitching a ride on a train, piggyback style.

    @Intercourse said:

    Also, a lot of their parcels carried by air in to remote areas (Alaska, Upper Peninsula, etc) are flown by independent carriers.

    Yeah, UPS and FedEx don't bother flying little planes to little airports; they leave that to cargo feederliner carriers (see FedEx Feeder for instance)



  • What's "DR"?


  • Grade A Premium Asshole

    @henke37 said:

    What's "DR"?

    Not sure if this is an honest question, and possibly setting myself up for a "Whoosh" badge, but DR is Disaster Recovery.


  • FoxDev

    $ wtf is DR
    DR: Disaster Recovery
    $
    


  • Also:

    • Dr. Repulsor
    • Dead Rising
    • A regular doctor


  • @Intercourse said:

    Disaster Recovery

    This is one of those things you have to have to check a box on the CIO's list of toys all my friends have so I want one too or I'm gonna throw the mother of all tantrums in the toy store. But no one actually tests it until you need it and that's when you find out that the backups are being piped to dev/nul and the CIO's intern nephew has been using the space to host the biggest warez site on the 'net.



  • My first thought was Dual Reverb but you'd have to be into Marshall guitar amps to think that way.

    Next up was Suzuki motorcycles, I think a bunch of them start with DR.


  • Grade A Premium Asshole

    @mott555 said:

    Next up was Suzuki motorcycles, I think a bunch of them start with DR.

    Their dirtbikes and dual-sports. It is a shame that the 2-stroke engine went the way of the dodo. I know that they pollute more, etc. Still, there is nothing like the fury of a big 2-stroke engine coming in to its power band. My first "real" dirtbike was an RM125 and when you are 12-13 years old it seemed like a rocket ship...that would do wheelies.



  • I've never ridden a 2-stroke, but every dirt bike I've owned has been a dual-sport so they're subject to emissions requirements for road vehicles.


  • FoxDev

    ok, so i hand typed that one. out of curriosity i tried it on an actual command line:

    accalia@Devbox2:$ wtf is DR
    Gee...  I don't know what DR means...
    accalia@Devbox2:$ wtf is IBM
    Gee...  I don't know what IBM means...
    accalia@Devbox2:$ wtf is ASAP
    ASAP: as soon as possible
    accalia@Devbox2:$ wtf is JDGI
    JDGI: Jeff Doesn't Get It
    accalia@Devbox2:$ 
    

    :blink:

    wait.....

    what was that last one again?

    accalia@Devbox2:$ wtf is JDGI
    JDGI: Jeff Doesn't Get It
    accalia@Devbox2:$ 
    

    who else here is a TDWTFer, cause i didn't add that one to the acronym database on our devbox...



  • @abarker said:

    UPS does outsource door deliveries.

    Sure, but it's not the same as Weng's situation. UPS doesn't just say "let's have someone else deliver our packages". They deliver most of their packages and use other providers for servicing unusually large loads (like holidays), areas that they can't service well, or to provide services that UPS doesn't do at the moment. The Post Office allows UPS to get into the small package business at around the five dollar price that UPS just doesn't have the systems in place to do.

    There are examples that are a bit more extreme. Netflix outsources all of its IT to Amazon. They seem like a data delivery company, but their primary business is negotiating contracts with movie studios. They don't succeed because they have a better streaming service, they succeed because they have better content. I'm sure when the market gets more crowded and prices get pushed down, Netflix will look into building data centers - or lose out to Apple, Microsoft, and Amazon due to not being able to compete on a level playing field.


  • 🚽 Regular

    @Intercourse said:

    there is nothing like the fury of a big 2-stroke engine coming in to its power band

    I accepted my father's offer for a ride on the back of his RD400 once. We tootled around a bit I was quite impressed with the feeling of power from only 400cc. Then we hit the dual-carriageway and he opened the taps, I honestly thought I was going to come off the back in spite of holding on for dear life.
    Power band is a misnomer on those bikes, sounds too nice, it should be called a power discontinuity.


  • Grade A Premium Asshole

    @Jaime said:

    Netflix outsources all of its IT to Amazon.

    Which is kind of amusing as Amazon Prime is likely one of their biggest competitors.

    @Jaime said:

    they succeed because they have better content.

    Meh, that depends on your preferences I suppose. We have been considering dropping their service as we cannot find much on their streaming service that we want to watch. Occasionally we might find some older movies (I saw "Planes, Trains and Automobiles" on there the other night), but for the most part we just can't find much we want to watch on there.


  • Grade A Premium Asshole

    @Cursorkeys said:

    Power band is a misnomer on those bikes, sounds too nice, it should be called a power discontinuity.

    You may be right about that, but I would bet that everyone remembers the first time they ride a 2-stroke. You are putzing around at low RPM, getting used to things and then you hit the throttle. For a moment you think, "What's the big deal?" Then it comes in to its own and HOLY SHIT! The front wheel comes off the ground, you hold of for dear life (usually pinning the throttle with your locked wrist) and it only calms down when it hits redline.

    My bike before that was an XR80. You would think an RM125 would only have 50% more power but it seemed like many orders of magnitude in difference. I have fond memories of that bike. 😄



  • @Intercourse said:

    You are putzing around at low RPM, getting used to things and then you hit the throttle. For a moment you think, "What's the big deal?" Then it comes in to its own and HOLY SHIT!

    You guys should try a modern 600cc sportbike. They're docile under 6000 rpm, but hit like a two stroke right around 10k. Oh, and they make more power in the part where they're described as "docile" than an RD400 makes at peak.

    I do know what you are talking about, I owned an RZ350 back in the 1990s. Trust me, what the kids today call "starter bikes", things like the ZX636, make more power than everything on the market in the 1980s, even legends like the H1 or the original Ninja 900.


  • Grade A Premium Asshole

    I am talking about a mid-90's dirtbike though. Sport bikes have made leaps and bounds in the past 10-15 years, no doubt. But I am referring to doing so at 12-13 years old on dirt or a gravel road. It seemed like a rocket at the time.

    The last really fast sport bike that I rode was probably a GXR750 that had a big bore kit and fuel injection added to it. Probably nothing compared to a bone stock bike today, but god it seemed fast at the time. It was faster than the first R1's and R6's.

    I have always wanted to get another bike, but now with a child and responsibilities...



  • Sportbike posture and I do not get along. I have a mild lower back injury from a car accident as a kid. Dirt bikes are the only category of motorcycle I've found that don't cause constant pain after 30 minutes of riding. They have plenty of legroom and an upright posture. Cruisers are the worst in my experience.

    Unfortunately most dual-sports top out around 250cc these days. There are a few in the 650cc range but the new generation of water-cooled 250's are about the same power as the older air-cooled 650's.



  • @Intercourse said:

    The last really fast sport bike that I rode was probably a GXR750 that had a big bore kit and fuel injection added to it.

    I just got rid of a Hayabusa with a big bore kit that made 200HP at the rear wheel. It did wheelies at 150mph and had a top speed of 205. Since I have a family now too, I'm moving on to a car with a 1:5 power to weight ratio.


  • Grade A Premium Asshole

    @Jaime said:

    Since I have a family now too, I'm moving on to a car with a 1:5 power to weight ratio.

    Probably a good call...


  • Garbage Person

    I've been wanting to get a motorcycle license and something like a CB500F. But I'm afraid that a 280lb man like myself would look utterly ridiculous.


  • Grade A Premium Asshole

    @Weng said:

    But I'm afraid that a 280lb man like myself would look utterly ridiculous.



  • I looked into the CB500F last time I was motorcycle shopping since I no longer lived in the boonies and some kind of street bike would be a logical choice, but since it's classed as a sport bike the insurance rates were far higher than I wanted to pay. My CRF250L costs next to nothing to insure, but still has a sport bike engine and transmission originally from the CBR250R model.



  • I paid $350 a year to insure my Hayabusa with full collision.


Log in to reply