But we have to stick to our budget!



  • Back in January I posted an article mentioning that we had already spent our budget for new disks for the year and that we were due to run out of disk space in March. We got an emergency infusion and the disks got installed a day or so before the system was "full".

    About 30 minutes ago, the operators got a warning from Oracle that the database was running out of space. Some quick math showed that we'll run out of space in 4 weeks.

    Emails fly. Conference calls abound. Fingers point.

    Why didn't the developers do something to reduce the amount of space we're consuming? Erm, we DID, but were told not to do it by SomePHB; here's the email!

    Ok, so can you do it now? Well, if we dropped everything else, we might be able to get it tested and deployed in time, but you'll still need to buy more disk space. Our change slows the rate of growth; it does not stop it.

    But there is no money in the budget!

    Fine; when we run out of space, the system will simply crash - hard - and stop.

    But what about our customers?

    Well it seems you have a choice: let us make the changes in full-panic-mode AND spend money on disks right now and stay in business, or shut down until the new year and piss off your customers and lose all your employees.

    We won't piss off the employees; there's money in the budget for payroll. So we'll all be sitting here writing code but no way to run it in production because the database is out of space?

    OBTW: we warned all of you about this last year, and nobody took us seriously. Now maybe you'll listen to us!

    But we have to stick to our budget!

     



  • You forgot the third option: run a DELETE statement without a WHERE clause.

    Can I haz monye now?



  • @C-Octothorpe said:

    run a DELETE statement without a WHERE clause.
    You're not far off the mark.

    They have us saving data in tabular form, and then the whole thing again as an xml string. A full 50% of the space is used by the table that has the xml strings. We could drop that table, instantly reclaim 50% of the space (>20T), and have anyone who needs the xml simply either a) query the tabular data directly, or b) we could write a service that wraps that action and return the xml string. Either way, it'd be on-the-fly.

    Our busniess users can't wrap their heads around the fact that data can be stored in forms other than xml, so they are just sitting on the problem instead of letting us solve it in a sensible way.

    At this point, we have queries to pull the data from the tabular tables, but we'd have to rehash a lot of code to pull the xml-generation functions out of where they are and make it callable from a service.



  • @snoofle said:

    But there is no money in the budget!

    Not really your problem. Unless your paychecks stop coming.

    Your resume is up to date, right?


  • @snoofle said:

    @C-Octothorpe said:

    run a DELETE statement without a WHERE clause.
    You're not far off the mark.

    They have us saving data in tabular form, and then the whole thing again as an xml string. A full 50% of the space is used by the table that has the xml strings. We could drop that table, instantly reclaim 50% of the space (>20T), and have anyone who needs the xml simply either a) query the tabular data directly, or b) we could write a service that wraps that action and return the xml string. Either way, it'd be on-the-fly.

    Our busniess users can't wrap their heads around the fact that data can be stored in forms other than xml, so they are just sitting on the problem instead of letting us solve it in a sensible way.

    At this point, we have queries to pull the data from the tabular tables, but we'd have to rehash a lot of code to pull the xml-generation functions out of where they are and make it callable from a service.

    So, 50% of storage is being eaten up by calculated data?  Why is the business involved in these technical details?  Shouldn't all IT just be some form of black magic or trickery?  Just tell them "you know what, don't worry about it, we'll take care of it."  If anything fails or goes sideways, you're responsible, as one would expect.


  • Well, it sounds like, in four weeks, you can let your DBAs upgrade your Oracle RAC setup from 10g to 11g. It'll only take a week of downtime, and you'll have three months of that until next year's budget gets you more disk space.



  • @C-Octothorpe said:

    Why is the business involved in these technical details?
    Politics, and it's not going to change. The IT brass kiss the business' ass on a routine basis, so the business gets to call the shots on budgets, and whether or not we can delete data. WE get to decide how to lay out new data, but THEY must be consulted before deleting old (even duplicated) data. The folks who are now business users were the ones who laid out the tables in the first place; they thought that the tabular data needed to be saved for (who knows why) purposes but figured that the XML would be used for actual retrieval. Stupid? Absolutely, but dropping production tables is something the DBAs simply will not do without brass signoff.

    Our hands are tied.

    If it wasn't so sad it'd be hysterical.

     



  • @snoofle said:

    But we have to stick to our budget!

    What's the over-under on how long this bout of cognitive dissonance will last? Have the C-levels gotten involved yet?



  • Honestly, if I didn't work where I worked -- where money/profit ISN'T the bottom line -- and see all the douchebaggery that happens here, I would seriously doubt that any of what you wrote is true.

    Alas...



  • @snoofle said:

    A full 50% of the space is used by the table that has the xml strings. ... At this point, we have queries to pull the data from the tabular tables, but we'd have to rehash a lot of code to pull the xml-generation functions out of where they are and make it callable from a service.
     

    I don't suppose Oracle has some kind of transparent compression option that you can turn on for text-heavy tables? XML usually deflates pretty well.

     



  • @flabdablet said:

    I don't suppose Oracle has some kind of transparent compression option that you can turn on for text-heavy tables?
     

    It does, but:

    1. it's applied to tablespaces, not tables - so will compress all objects in that tablespace (tables, views, indexes, etc)
    2. compression applies to data blocks as they're written to the tablespace, not to existing data in that tablespace.
    Retrospective compression simply requires creating a new (compressed) tablespace then moving objects (tables) out of their current location and into the new one. It's all transparent to users. 



  • @Cassidy said:

    Retrospective compression simply requires creating a new (compressed) tablespace then moving objects (tables) out of their current location and into the new one.

    That should work. Unless something crazy was happening, like maybe running out of disk space.

    Sigh.

     



  • @flabdablet said:

    @Cassidy said:

    Retrospective compression simply requires creating a new (compressed) tablespace then moving objects (tables) out of their current location and into the new one.

    That should work. Unless something crazy was happening, like maybe running out of disk space.

    Sigh.

     

    DRVSPACE!

    Edit: also ramdrives!



  • @flabdablet said:

    That should work. Unless something crazy was happening, like maybe running out of disk space.

    Sigh. 

    I know that one. A client's disks were full and my colleagues had delivered a system that didn't delete old data. Performance was obviously also lousy. Not that it was the fault of my colleagues: the client's internal politics made it impossible for them to decide on which data to keep and which to delete.

    Spicy aside: a lot of that data should be deleted according to by their own privacy regulations, so there really was nothing to discuss.

    Anyway, when disk space was really getting tight, one of my colleagues ended up moving data from old to new tables, one month at a time, backing it up, deleting it, etc., until there was some room to breath and do things the proper way. I think it took him over a week of making small changes to scripts and watching over the wretched system that nothing would go wrong...



  • @flabdablet said:

    @Cassidy said:

    Retrospective compression simply requires creating a new (compressed) tablespace then moving objects (tables) out of their current location and into the new one.

    That should work.

     

    .. unless, also, they're doing it already.

    Or their DBAs are unaware compression is even an option.

    Given snoofle's past reports on the attitude and behaviour of their DBAs, I'm inclined to believe the latter. There does seem to be a big "not too proactive, don't really care" culture there.

     



  • Won't compression add to their CPU overheating woes?



  • @Cassidy: compression...

    Our DBAs actually know that 11g supports compression, although they have no experience in using it.

    Our DB is currently 10g, and compression would have to be done with lz_compress... Not really an option given the time frame. It was decided to wait for 11g with built-in compression. The problem is, as mentioned, that we're out of space, so creating a parallel table that consumes 20+T in order to compress existing data is - well - just not going to happen.

    What's going to happen is that at the very last second, they'll scream: create a wrapper service and drop the table. Then we'll have to "scramble". I've already written and tested a wrapper service - under the radar. Now, when it hits the fan, I won't need to run around like a chicken without a head - because I planned ahead.



  • @snoofle said:

    Now, when it hits the fan, I won't need to run around like a chicken without a head - because I planned ahead.

    ENABLER!



  • @Zemm said:

    Won't compression add to their CPU overheating woes?

    In the same way that a sticker depicting the fuel consumption rate of a car actually changes the rate when attached to the car's windscreen.



  • @snoofle said:

    Now, when it hits the fan, I'll give the impression I'm running around like a headless chicken - because I aim to milk their crap decision-making process for all it's worth.
     

    snoofled that for you.

    Really, isn't this a DBA problem? They're running short of capacity, it's up to them to sort it out? I thought your remit was middleware that interracts with the back-end DBs.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.