But we have to stick to our budget!



  • Back in January I posted an article mentioning that we had already spent our budget for new disks for the year and that we were due to run out of disk space in March. We got an emergency infusion and the disks got installed a day or so before the system was "full".

    About 30 minutes ago, the operators got a warning from Oracle that the database was running out of space. Some quick math showed that we'll run out of space in 4 weeks.

    Emails fly. Conference calls abound. Fingers point.

    Why didn't the developers do something to reduce the amount of space we're consuming? Erm, we DID, but were told not to do it by SomePHB; here's the email!

    Ok, so can you do it now? Well, if we dropped everything else, we might be able to get it tested and deployed in time, but you'll still need to buy more disk space. Our change slows the rate of growth; it does not stop it.

    But there is no money in the budget!

    Fine; when we run out of space, the system will simply crash - hard - and stop.

    But what about our customers?

    Well it seems you have a choice: let us make the changes in full-panic-mode AND spend money on disks *right now* and stay in business, or shut down until the new year and piss off your customers and lose all your employees.

    We won't piss off the employees; there's money in the budget for payroll. So we'll all be sitting here writing code but no way to run it in production because the database is out of space?

    OBTW: we warned all of you about this last year, and nobody took us seriously. Now maybe you'll listen to us!

    But we have to stick to our budget!

     



  • You forgot the third option: run a DELETE statement without a WHERE clause.

    Can I haz monye now?



  • @C-Octothorpe said:

    run a DELETE statement without a WHERE clause.
    You're not far off the mark.

    They have us saving data in tabular form, and then the whole thing again as an xml string. A full 50% of the space is used by the table that has the xml strings. We could drop that table, instantly reclaim 50% of the space (>20T), and have anyone who needs the xml simply either a) query the tabular data directly, or b) we could write a service that wraps that action and return the xml string. Either way, it'd be on-the-fly.

    Our busniess users can't wrap their heads around the fact that data can be stored in forms other than xml, so they are just sitting on the problem instead of letting us solve it in a sensible way.

    At this point, we have queries to pull the data from the tabular tables, but we'd have to rehash a lot of code to pull the xml-generation functions out of where they are and make it callable from a service.



  • @snoofle said:

    But there is no money in the budget!

    Not really your problem. Unless your paychecks stop coming.

    Your resume is up to date, right?


  • @snoofle said:

    @C-Octothorpe said:

    run a DELETE statement without a WHERE clause.
    You're not far off the mark.

    They have us saving data in tabular form, and then the whole thing again as an xml string. A full 50% of the space is used by the table that has the xml strings. We could drop that table, instantly reclaim 50% of the space (>20T), and have anyone who needs the xml simply either a) query the tabular data directly, or b) we could write a service that wraps that action and return the xml string. Either way, it'd be on-the-fly.

    Our busniess users can't wrap their heads around the fact that data can be stored in forms other than xml, so they are just sitting on the problem instead of letting us solve it in a sensible way.

    At this point, we have queries to pull the data from the tabular tables, but we'd have to rehash a lot of code to pull the xml-generation functions out of where they are and make it callable from a service.

    So, 50% of storage is being eaten up by calculated data?  Why is the business involved in these technical details?  Shouldn't all IT just be some form of black magic or trickery?  Just tell them "you know what, don't worry about it, we'll take care of it."  If anything fails or goes sideways, you're responsible, as one would expect.


  • Well, it sounds like, in four weeks, you can let your DBAs upgrade your Oracle RAC setup from 10g to 11g. It'll only take a week of downtime, and you'll have three months of that until next year's budget gets you more disk space.



  • @C-Octothorpe said:

    Why is the business involved in these technical details?
    Politics, and it's not going to change. The IT brass kiss the business' ass on a routine basis, so the business gets to call the shots on budgets, and whether or not we can delete data. WE get to decide how to lay out new data, but THEY must be consulted before deleting old (even duplicated) data. The folks who are now business users were the ones who laid out the tables in the first place; they thought that the tabular data needed to be saved for (who knows why) purposes but figured that the XML would be used for actual retrieval. Stupid? Absolutely, but dropping production tables is something the DBAs simply will not do without brass signoff.

    Our hands are tied.

    If it wasn't so sad it'd be hysterical.

     


  • ♿ (Parody)

    @snoofle said:

    But we have to stick to our budget!

    What's the over-under on how long this bout of cognitive dissonance will last? Have the C-levels gotten involved yet?



  • Honestly, if I didn't work where I worked -- where money/profit ISN'T the bottom line -- and see all the douchebaggery that happens here, I would seriously doubt that any of what you wrote is true.

    Alas...



  • @snoofle said:

    A full 50% of the space is used by the table that has the xml strings. ... At this point, we have queries to pull the data from the tabular tables, but we'd have to rehash a lot of code to pull the xml-generation functions out of where they are and make it callable from a service.
     

    I don't suppose Oracle has some kind of transparent compression option that you can turn on for text-heavy tables? XML usually deflates pretty well.

     



  • @flabdablet said:

    I don't suppose Oracle has some kind of transparent compression option that you can turn on for text-heavy tables?
     

    It does, but:

    1. it's applied to tablespaces, not tables - so will compress all objects in that tablespace (tables, views, indexes, etc)
    2. compression applies to data blocks as they're written to the tablespace, not to existing data in that tablespace.
    Retrospective compression simply requires creating a new (compressed) tablespace then moving objects (tables) out of their current location and into the new one. It's all transparent to users. 


  • @Cassidy said:

    Retrospective compression simply requires creating a new (compressed) tablespace then moving objects (tables) out of their current location and into the new one.

    That should work. Unless something crazy was happening, like maybe running out of disk space.

    Sigh.

     



  • @flabdablet said:

    @Cassidy said:

    Retrospective compression simply requires creating a new (compressed) tablespace then moving objects (tables) out of their current location and into the new one.

    That should work. Unless something crazy was happening, like maybe running out of disk space.

    Sigh.

     

    DRVSPACE!

    Edit: also ramdrives!



  • @flabdablet said:

    That should work. Unless something crazy was happening, like maybe running out of disk space.

    Sigh. 

    I know that one. A client's disks were full and my colleagues had delivered a system that didn't delete old data. Performance was obviously also lousy. Not that it was the fault of my colleagues: the client's internal politics made it impossible for them to decide on which data to keep and which to delete.

    Spicy aside: a lot of that data should be deleted according to by their own privacy regulations, so there really was nothing to discuss.

    Anyway, when disk space was really getting tight, one of my colleagues ended up moving data from old to new tables, one month at a time, backing it up, deleting it, etc., until there was some room to breath and do things the proper way. I think it took him over a week of making small changes to scripts and watching over the wretched system that nothing would go wrong...



  • @flabdablet said:

    @Cassidy said:

    Retrospective compression simply requires creating a new (compressed) tablespace then moving objects (tables) out of their current location and into the new one.

    That should work.

     

    .. unless, also, they're doing it already.

    Or their DBAs are unaware compression is even an option.

    Given snoofle's past reports on the attitude and behaviour of their DBAs, I'm inclined to believe the latter. There does seem to be a big "not too proactive, don't really care" culture there.

     



  • Won't compression add to their CPU overheating woes?



  • @Cassidy: compression...

    Our DBAs actually know that 11g supports compression, although they have no experience in using it.

    Our DB is currently 10g, and compression would have to be done with lz_compress... Not really an option given the time frame. It was decided to wait for 11g with built-in compression. The problem is, as mentioned, that we're out of space, so creating a parallel table that consumes 20+T in order to compress existing data is - well - just not going to happen.

    What's going to happen is that at the very last second, they'll scream: create a wrapper service and drop the table. Then we'll have to "scramble". I've already written and tested a wrapper service - under the radar. Now, when it hits the fan, I won't need to run around like a chicken without a head - because I planned ahead.

  • ♿ (Parody)

    @snoofle said:

    Now, when it hits the fan, I won't need to run around like a chicken without a head - because I planned ahead.

    ENABLER!



  • @Zemm said:

    Won't compression add to their CPU overheating woes?

    In the same way that a sticker depicting the fuel consumption rate of a car actually changes the rate when attached to the car's windscreen.



  • @snoofle said:

    Now, when it hits the fan, I'll give the impression I'm running around like a headless chicken - because I aim to milk their crap decision-making process for all it's worth.
     

    snoofled that for you.

    Really, isn't this a DBA problem? They're running short of capacity, it's up to them to sort it out? I thought your remit was middleware that interracts with the back-end DBs.



  • @Cassidy said:

    @snoofle said:

    Now, when it hits the fan, I'll give the impression I'm running around like a headless chicken - because I aim to milk their crap decision-making process for all it's worth.
     

    snoofled that for you.

    Really, isn't this a DBA problem? They're running short of capacity, it's up to them to sort it out? I thought your remit was middleware that interracts with the back-end DBs.

    It is up to the DBAs to supply enough disk space, but the simple fact remains that the folks who intially designed and wrote our system dumped way too much crap to the database.

    The software I wrote is simply a wrapper around a few queries to dynamically generate the xml so it can be provided to all the other systems (otherwise, multiple systems would need to change - in very short order - to handle the raw data). It's a complex, but easy-to-understand user (db) function.

    We can then create a view that calls my user function for any queried rows. This way, none of the systems that rely on the xml table will know anything has changed, and can be migrated over time.

    Someone else will need to write the web service and make the single call to my user function. As I understand it, writing a web service with a single entry point is a fairly simple deal.

    Stupid? Absolutely. But it gets them out of the time/space jam (which is kind of my job) and gives them time to fix it properly.

    Although..... I can see more WTFs coming from this approach; the folks that consume this data wrote their own XML parser, and in their world, all spaces are significant.

     



  • @snoofle said:

    It is up to the DBAs to supply enough disk space, but the simple fact remains that the folks who intially designed and wrote our system dumped way too much crap to the database.
     

    Yeah, but I'm arguing that (a) the DBAs are responsible for monitoring consumption and raising potential capacity-related issues, and (b) the DBAs should be planning and driving change to provide sufficient capacity at database level, irrespective of how it's used.

     Or more succintly: it's an issue that affects them, it should be THEIR job to push it back and/or fix it... not yours (you're not the DBA team, are you?). Let the DBAs and the budget-holders fight it out, just stand well back.

    (yes, I know loss of DB affects every user of the service - but it just sounds like you're taking ownership of a problem that you're not accountable for nor are you responsible for fixing).



  • @Evilweasil said:

    Your resume is up to date, right?
     

    No Snoofle, don't quit! Your stories are magnificent!

    We'll take up a collection and hire you as an undercover reporter.

     

     


  • ♿ (Parody)

    @Cassidy said:

    Yeah, but I'm arguing that (a) the DBAs are responsible for monitoring consumption and raising potential capacity-related issues, and (b) the DBAs should be planning and driving change to provide sufficient capacity at database level, irrespective of how it's used.

    Or more succintly: it's an issue that affects them, it should be THEIR job to push it back and/or fix it... not yours (you're not the DBA team, are you?). Let the DBAs and the budget-holders fight it out, just stand well back.

    Accountability games are fun and all, but it doesn't seem like there's anything they (DBAs) can actually do at to fix anything in the short / medium term.



  • @Cassidy said:

    @snoofle said:

    It is up to the DBAs to supply enough disk space, but the simple fact remains that the folks who intially designed and wrote our system dumped way too much crap to the database.
     

    Yeah, but I'm arguing that (a) the DBAs are responsible for monitoring consumption and raising potential capacity-related issues, and (b) the DBAs should be planning and driving change to provide sufficient capacity at database level, irrespective of how it's used.

     Or more succintly: it's an issue that affects them, it should be THEIR job to push it back and/or fix it... not yours (you're not the DBA team, are you?). Let the DBAs and the budget-holders fight it out, just stand well back.

    (yes, I know loss of DB affects every user of the service - but it just sounds like you're taking ownership of a problem that you're not accountable for nor are you responsible for fixing).

    Sorry, I misunderstood what you were going for. I agree completely. Sadly, the DBAs do not do capacity planning around here. They should, but they don't; it falls to us. If WE don't warn the brass, then it's OUR fault, and it doesn't matter how many times we tell them that the DBAs should be on this, or that our calcs show we're going to run dry by <date>, they just sit on it - because <I don't know why>.

    At least, thanks to Alex, I have an outlet.

     



  • All: as I was writing that last reply to Cassidy, I got an email from a friend on the corporate side of things. He's on the budget sub committee of their capacity planning team, and heard - through the grapevine- about our storage/budgetary issues. I called him and explained everything (no way I'm putting that in writing, as I'd be bypassing 4 levels of management).

    He informed me that there's a capital budget for unforseen stuff - on the corporate side - and it could be used to fund our immediate need. He could also push from his side to force the brass to reprioritize and let us fix the bleeding.

    Now all I have to do is figure out how to tell the brass that I found out about this and that all they need to do is ask - without also letting them know I blabbed over their heads.

    Ideas?

     



  • @snoofle said:

    Now all I have to do is figure out how to tell the brass that I found out about this and that all they need to do is ask - without also letting them know I blabbed over their heads.

    Ideas?

    Seems like the best solution is for every management layer between you and "the corporate side of things" needs to be thrown out on the street. Or reassigned to coffee maker duty.



  • @blakeyrat said:

    the best solution is for every management layer between you and "the corporate side of things" needs to be thrown out on the street
    True, but unlikely.



  • @snoofle said:

    Now all I have to do is figure out how to tell the brass that I found out about this and that all they need to do is ask - without also letting them know I blabbed over their heads.

    Ideas?

     

    Just go in there as if it's common knowledge.

    "Why not use the capital budget for unforseen expenditure to increase disk space?"
    "How do you know about that?"
    "In a company of this size? How do you not know about that?"



  • @snoofle said:

    All: as I was writing that last reply to Cassidy, I got an email from a friend on the corporate side of things. He's on the budget sub committee of their capacity planning team, and heard - through the grapevine- about our storage/budgetary issues. I called him and explained everything (no way I'm putting that in writing, as I'd be bypassing 4 levels of management).

    He informed me that there's a capital budget for unforseen stuff - on the corporate side - and it could be used to fund our immediate need. He could also push from his side to force the brass to reprioritize and let us fix the bleeding.

    Now all I have to do is figure out how to tell the brass that I found out about this and that all they need to do is ask - without also letting them know I blabbed over their heads.

    Ideas?

     Tell them you read about it on Wikileaks. Report back on their response.



  • @timbstoke said:

    @snoofle said:

    Now all I have to do is figure out how to tell the brass that I found out about this and that all they need to do is ask - without also letting them know I blabbed over their heads.

    Ideas?

    Just go in there as if it's common knowledge.

    "Why not use the capital budget for unforseen expenditure to increase disk space?"
    "How do you know about that?"
    "In a company of this size? How do you not know about that?"

    That, or just ask Mr. Higher-Up Corporate Guy to instigate the conversation from his end, because it would be awkward if you did it from your end.

     



  • @snoofle said:

    A full 50% of the space is used by the table that has the xml strings. We could drop that table, instantly reclaim 50% of the space (>20T),

     

     

    TRWTF

     



  • @boomzilla said:

    What's the over-under on how long this bout of cognitive dissonance will last?

    The problem here isn't cognitive dissonance, but rather the ABSENCE of cognitive dissonance. Cognitive dissonance is good-- it's the brain's way of letting you know you're trying to hold simultaneously conflicting thoughts.

     


  • ♿ (Parody)

    @Zylon said:

    @boomzilla said:
    What's the over-under on how long this bout of cognitive dissonance will last?

    The problem here isn't cognitive dissonance, but rather the ABSENCE of cognitive dissonance. Cognitive dissonance is good-- it's the brain's way of letting you know you're trying to hold simultaneously conflicting thoughts.

    Yes, and I think that they know that something is wrong, and that something must be done. They just haven't yet decided if they're going to screw their customers or their budget. Well, maybe the double secret emergency budget will let them save both in the end.



  • @Mason Wheeler said:

    ask Mr. Higher-Up Corporate Guy to instigate the conversation from his end, because it would be awkward if you did it from your end.
    They must have read your thoughts because an inquiry came down while I was at lunch. It was cc'd to 5 levels of management, from my boss up to C*. It was lambasting in nature. It demanded to know how management could let a business-critical function get to this state without acting upon it.It ordered a conference call just after lunch. My boss wants me in on it.

    I'm spending the next few minutes hunting down and flagging all the emails we sent warning of impending doom and begging for resources, so we can forward them to everyone on the call as soon as someone says that development never raised the flag.

    I'm betting that by the end of the day, the corporate capacity planning folks will force the issue, and money will be made available.

    I'll let you know.

     



  • @boomzilla said:

    double secret emergency budget
    Sadly, before the parent company bought us, when we needed to get stuff, it just involved walking over to a 3rd level manager, giving a quick explanation of what, why and how much, and you were given the verbal go-ahead to get whatever is was. Now, the parent company is forcing budgeting procedures down our throats, and threatening managers who go over budget without following procedures.

    True, if proper capacity planning was done by the folks who should be doing it (e.g.: DBAs in this case), that the red flags could have gone up in time for budgeting to be put in place so stuff could be planned, bought and installed before stuff blew up.  This particular (storage) problem has been festering for 18 months. My boss raised the red flag at that time. I raised the red flag as soon as I found out about it (a year ago).



  • @timbstoke said:

    Just go in there as if it's common knowledge.

    "Why not use the capital budget for unforseen expenditure to increase disk space?"
    "How do you know about that?"
    "In a company of this size? How do you not know about that?"

     

    +1.

    Make out that you're aware the company is good at contingency planning, and this is JUST the sort of thing they'd planned for, thank $deity. The customers WILL be relieved!



  • @snoofle said:

    True, if proper capacity planning was done by the folks who should be doing it (e.g.: DBAs in this case), that the red flags could have gone up in time for budgeting to be put in place so stuff could be planned, bought and installed before stuff blew up.  This particular (storage) problem has been festering for 18 months. My boss raised the red flag at that time. I raised the red flag as soon as I found out about it (a year ago).
     

    Can this be flagged up now?

    Not the actual situation, but how the situation arose: those who should have red-flagged and avoided the situation don't see it as their responsibility, andnow we've had to dip into the DBA's pay packets emergency fund unnecesarily.



  • @snoofle said:

    Conference call with corporate
     

    Corporate: The brass introduced themselves and demanded to know how this situation could not be forseen, and why nothing was done about it.

    Managers (ours): But development brought it up to us; we just didn't have the money in the budget, so we left it on the back burner!

    C: When were you told of this issue

    My boss nudges me so I chime in: We knew about it 18 months ago; I'm forwarding the relevant emails to everyone now

    In the background, I hear new-mail dings, and soft whispers that I'll politely summarize as: Are you kidding me?

    C: The budget is only an estimate; when emergencies come up, you raise the issue up

    M: Even if we had the money, we are working on a top priority performance fix for customer X

    C: Put it on the back burner: if the database runs out of space, every customers' performance will go to ZERO - do you want THAT to happen?

    M: um...

    C: (To my boss): Assuming you had the money, how quickly are you consuming disk space

    B:  We're using x TB/month, with expected growth of Y due to xxx initiative. We have a plan in place to slow the rate of consumption, but it's on the back burner due to <other priorities>. If we were permitted, we could divert 3 developers and get this done and tested  in 2 weeks. Once done, we'll slow our growth rate from x TB/month to 0.85x, and we can then drop the 20TB table which will give us substantial leeway. What we lack is permission

    C: (To the purchasing folks):  get two months of disks in here ASAP as a hedge. 

    C: (To my boss): put your three best folks on this immediately and get it done; keep everyone apprised of progress/problems

    C: (To senior managers on our side): we need to have a meeting to discuss this in further detail

    The corporate folks seem to get things. It's just that our managers haven't yet drank the kool-ade, and don't know how to play inside the coporate sandbox; they're still acting like we're a new startup, but with our hands tied by our coporate masters.

    I showed my boss what I had working so far. He grabbed one of our web-literate folks who said he could have the web side done and tested in a day.  Our team DBA (not the cross-team DBAs - who are the source of most of these WTFs) said he could have the view we need up and running in an hour.

    My boss figures we can have this done by tomorrow, and promised that if we do, he's buying the drinks tomorrow night.

     



  • Sounds like Mr. Higher-Up Guy has a both good head on his shoulders, and the authority to do worthwhile things with it.  Try sending him the following link, explaining that it probably describes the problem fairly well, and invite him to figure out where the "thermocline layer" exists:

    http://brucefwebster.com/2008/04/15/the-wetware-crisis-the-themocline-of-truth/

    (This is worth should be required reading for anyone in an organization with more than two layers separating you (the grunts) from the CEO.)



  • @Mason Wheeler said:

    Sounds like Mr. Higher-Up Guy has a both good head on his shoulders, and the authority to do worthwhile things with it. 
     

    That.

    Also sounds like some middle-management are due for a session with the Spiky Purple Dildo of Refocussed Priorities.

    I'll be interested to hear if any DBAs experience the Clue By Four of Responsibility Recognition.



  • @Mason Wheeler said:

    figure out where the "thermocline layer" exists:
    Very interesting read - I can relate after nearly 30 years of big-corp projects.

    We know EXACTLY where and whom the thermocline layer is; it's my boss+1. Sending memos to this person is like: echo "Warning: ..." > /dev/nul

    Frequently, My boss+2 will grab my boss and ask what's actually happening.

    Unfortunately, while Boss+1 will be vindictive and fire someone who pisses him off, Boss and Boss+2 won't fire anybody unless they've committed some mortal-sin; give them another chance; work with them!

     



  • @Cassidy said:

    @Mason Wheeler said:

    Sounds like Mr. Higher-Up Guy has a both good head on his shoulders, and the authority to do worthwhile things with it. 
     

    That.

    Also sounds like some middle-management are due for a session with the Spiky Purple Dildo of Refocussed Priorities.

    I'll be interested to hear if any DBAs experience the Clue By Four of Responsibility Recognition.

    Is that anything like Evard's Spiked Tentacles of Forced Intrusion?

     



  • @snoofle said:

    @Mason Wheeler said:

    figure out where the "thermocline layer" exists:
    Very interesting read - I can relate after nearly 30 years of big-corp projects.

    We know EXACTLY where and whom the thermocline layer is; it's my boss+1. Sending memos to this person is like: echo "Warning: ..." > /dev/nul

    Frequently, My boss+2 will grab my boss and ask what's actually happening.

    Unfortunately, while Boss+1 will be vindictive and fire someone who pisses him off, Boss and Boss+2 won't fire anybody unless they've committed some mortal-sin; give them another chance; work with them!

    And I don't suppose telling Boss +5 (the guy you've been talking about who's actually getting stuff done) that boss +1 has consistently been the source of problems like this would accomplish anything?

     



  • @Cassidy said:

    Also sounds like some middle-management are due for a session with the Spiky Purple Dildo of Refocussed Priorities.

    The corporate brass wants a face-to-face sit-down with our Boss+2...C*, so I imagine that this may well happen

    @Cassidy said:

    I'll be interested to hear if any DBAs experience the Clue By Four of Responsibility Recognition.

    Not gonna happen; these guys are grandfathered in by <don't know who>; they seem to be immune from criticism by anyone I've met here so far.

     




  • @Mason Wheeler said:

    And I don't suppose telling Boss +5 (the guy you've been talking about who's actually getting stuff done) that boss +1 has consistently been the source of problems like this would accomplish anything?
    Probably not; He's not Boss+5; he's not even in the hierarchy above me. He's on the budget subcommittee of the corporate capacity planning team. It's just that he realized that a very public shit-storm was about to occur, so he bucked it up his chain of management who reacted quickly.

    I sincerely doubt that Boss+1 will have any consequences, other than a lashing in the corp/local-boss meeting.



  •  ...on the upside, I made a whole bunch of friends in the capacity planning arena (for quietly raising a red flag). It might come in handy down the road.



  • @Cassidy said:

    Also sounds like some middle-management are due for a session with the Spiky Purple Dildo of Refocussed Priorities.
     

    Is that lighter or heavier than this?



  • @snoofle said:

    we could divert 3 developers and get this done and tested in 2 weeks.

    My boss figures we can have this done by tomorrow

    Ah, the good old "Take your estimate, double it, then bump it up to the next largest unit of measure" estimation method.

     

     



  • Please just use a truncate statement instead.  Otherwise your logs will grow as every row will be logged. 


Log in to reply