Representative conversation


  • Garbage Person

    A conversation with the boss I share with one of our alleged senior developers (unfortunately he's not in my chain of command - he's one of my boss' 75 directly reporting developers).

    Keep in mind that my team builds a business framework and this clod is on the team that builds applications to run in that framework.

    Weng: "I don't even understand what he's doing. I don't think he knows what <fundamental business term around which the entire platform is based> means. I can clearly see he started with the template code, but what grew up around it is utterly incomprehensible, totally nonstandard, and doesn't seem to fit any requirement that <project manager> wouldn't have tossed right back out on its ass. Can code have rabies? I think this has rabies. There's literally no physical way this can coexist with any other running jobs. I have to shut it down or we're going to blow even more SLA's"
    Boss: "So we need a rewrite."
    Weng: "No. We need a fucking exorcist. Now do you want to talk about how the last modified date on the code in prod is yesterday, but the last official rollout was two weeks ago?"
    Boss: "No. I've heard it all before."
    Weng: "Can I lock him out of production yet?"
    Boss: "No. He's in the on-call rotation."
    Weng: "So take him out. Fuck, as much extra work as he makes for me, I'll take his week."
    Boss: "No."

    Do you know how many SLA's we missed because of this? Neither do I. Processing is still catching up, so there might yet be more casualties. Best guess at the moment is somewhere in the vicinity of 1500.

    The truly ridiculous thing is that I think we're just going to skate on the whole goddamned fuckup, because the head of our support group was out today and his cover didn't follow the correct Major Incident Reporting procedure. Apparently if you don't report the major incident, it didn't happen. Or something.


  • BINNED

    @Weng said:

    75 directly reporting developers

    TRWTF


  • Garbage Person

    Yeah. It is. My business unit is headed by a VP (who personally reports to the CIO). He has ~20 direct reports, almost all Directors (there are a few Managers. Who direct report.)

    Each of those directors directly manages a single team of between 2 and 7 Developers. On teams with 7 developers, one of them is nominally a Manager, but in a sane universe would be a team lead.

    Except my branch. My boss is a director. He has 75 direct reports. ALL of them are developers (except the handful of PM's and Admin Assistants). Including me. Except, for some reason, I have developers direct-reporting to me. Which, in this structure, should logically make me a manager, but no, I'm apparently a team lead.

    Now, my branch was imported wholesale from another business unit and was only 15-strong at the start of last year, so some org chart absurdity is to be expected. But nobody has said a word about reorganizing it to the status quo for the business unit (which I am all in favor of, because I become a director and double my pay) or into some sort of sane structure.



  • @Weng said:

    which I am all in favor of, because I become a director and double my pay

    Yeah, right. If that happens they'll probably hire a new manager and allow you to keep on chugging release after release.


  • Garbage Person

    There's also an analogous "Developer" position in the same pay slot that 1) Isn't called Developer and 2) Is issued like candy as payoffs to keep critical personnel.

    And if they don't make with the payoffs soon, I'm out.



  • Been there, it's called "Architect".


  • Garbage Person

    Actually, I think it's "Senior Technical Analyst 5". There are no 1-4. (Developer stops at 3. Architect only exists at 3.)



  • At least it's not some lame "ninja", "guru", "evangelist" title bulshit.


  • Discourse touched me in a no-no place

    The #1 reason for having some sort of BS title is so that HR doesn't say “but this business unit over there pays a Software Engineer 15k so you should only be paid that much”. Yeah, that other business unit are a bunch of abusive WTFs (in one way or another, but principally for thinking they can get away with that sort of BS just because) but that doesn't mean that we want HR fucking things up in accordance with their long track record.


  • Discourse touched me in a no-no place

    I bet that guy got beat up a lot in high school.


  • Garbage Person

    So here's my personal favorite HR policy at this place. Only business units under the overarching banner of "Corporate IT" may hire developers. Since this was retroactive, CorpIT imported all the programmers from other business units and established new, independent units for each set, with their mandates limited to "what you do now and no more".

    So when a business unit finds a new need (say a line of business piece of scheduling software) they can't go to their former set of programmers and get it done. But they aren't allowed to hire programmers. Nor is there any process for getting anyone to do anything outside their mandate. So they hire programmers into customer service slots or whatever.

    But CorpIT got wise to this quickly, and banninated anyone not wearing a blessed job title from writing code in <list of all real development environments used companywide>.

    So they do their LOB apps in Access, Excel and Filemaker.

    Until they get caught. And then the Access, Excel or Filemaker pile of shit gets taken from them and given to a randomly selected CorpIT development group to maintain. But the business ownership stays with the business group, and rewriting it in anything not terrible is unacceptable risk.


  • Garbage Person

    So I'm in the office today (Sunday) hacking on this piece of shit app. Lights out. Speakers on loud. PA system hijacked (If I'm going to do this shit, not even having to walk to the bathroom is going to break my mental state, damn anyone else who thinks they're going to work). Pandora trance/industrial/dance mix very much on.

    Problem:
    The entire architecture of the app deviates from standards in such a way that I'd literally have to rewrite it from scratch. Given I don't have the requirements and cannot divine them from this fevered mass of Codethulhu, I ain't doin' that. Mind you I could do it in an afternoon, whereas the original development cycle was weeks.

    Solution:
    I'm going to double down on the crazy and OPTIMIZE this pile of WTF.

    I ran a representative file with 4 records in it. The bulk of the wait (discounting stuff that legitimately needs to take time) comes down to these 4 stored procedures:

    1. Get record count (WTF!?!?!?!?) - 3 minutes
    2. Get manufacturing bill of materials - 3 minutes
    3. Get list of documents groups we need to output - 13 minutes
    4. Get the list of documents we just output (there may be more than 1 document per group) - 5 minutes

    I'll tackle #3 first.


  • Garbage Person

    #3 (I hate markdown)

    It's 314 lines of irredeemable schlock. It's clearly based on the reference template code for getting this list, but with a number of customizations. The particular customizations present have their own unofficial reference implementations on this platform version and are actually so common that they're baked into the reference implementation on the newest revision of the platform. The implementations here are NOT EVEN CLOSE to any of those reference implementations.

    Takes 3 parameters. The manufacturing batch ID, a bool for whether this batch needs special handling, and a bool for whether we're in test mode or not.

    Outputs parameters that need to be later passed to the document composition software - one line for each group of documents.

    Has a 4-result nested if/else for the two bools. One bool basically controls what column a particular output value comes from. Yanked that if out and replaced it with an inline case statement.
    The other bool determines whether the second block in a UNION ALL is included. The 'true' branch for this contains the full union. The 'false' branch contains only the first block without the union. Yanked that and added an 'AND @var = 1' to the where on the second block in the union. No more if/else branching.

    That leaves us with the meat of things being two big ugly selects union all'd together.

    Each of those blocks contains a series of four joins on a bullshit Entity-Attribute-Value antipattern table (my crusade for the murder of this table was what brought me to prominence within the organization and its utter destruction in latter revisions of the platform is my most famous achievement - which is just fucking depressing) to get a series of values that are actually the same for every result in the table. Since retroactively destroying that table in previous versions of the platform has been held up in the works, as has its properly-normalized replacement, I'm stuck with it for the moment. But hey, the values are all the same.

    'SELECT TOP 1 @shittyvalue1=joina.value, @shittyvalue2 = joinb.value INTO #WTF FROM .... JOIN ..... JOIN.... JOIN..... JOIN..... JOIN.....'

    and remove the joins, using the variables instead.

    The second block in the union joins on a fixed, static value in a reference table. On further examination, that value is NOT the unique key to that reference table, but instead only part of a composite key matching 16 rows. All the values we're actually using are identical in all 16 rows. WTF. It all gets filtered out by a distinct. Anyway, "SELECT TOP 1 * INTO #wtf FROM referencetable WHERE partofthefuckingkey = 'idioticfixedvalue'" and replace the join with a cross join on #wtf.

    Joins on some tables that are never used. Axe those.

    Both blocks have a DISTINCT on them. What happens if I remove them?
    9200 results instead of 2. A DISTINCT is correct in this instance per the reference implementation, but the number of raw results should only be 1 per raw record (there are 4 in the test case). Apparently some of these customization tables have duplicate keys in them. I investigated, and it turns out we're only joining on partial keys again. Defective design. Reference data isn't normalized. Will have to leave it.

    It now takes 13 seconds to run, as opposed to 13 minutes. It's still a fucking abomination, but I think I can let it go now.

    End result is 137 lines, including the 21-line comment block that is 20 lines of 'WARNING!WARNING!'
    and 1 line of 'THIS IS NOT THE CORRECT WAY TO DO ANYTHING. ANYONE CAUGHT USING THIS AS REFERENCE MATERIAL WILL BE SUMMARILY EXECUTED'


  • Garbage Person

    Additional bonus wtf: One of the values in the result table is the name of a script file.

    In the reference versions, this is literally the name of the script file. Nothing more.
    In this version, it's the fully qualified name and location of the script file.

    Because the reference table it came from was copied from test, it contains the location of the script.... In the test environment. Since the interpreter is a very expensive third party product, it is shared between test and prod - so it happily pulls the test copy. Leaving that alone and simply alerting the authorities.


  • Garbage Person

    Next up: #4. My gut feeling here is that this can be replaced by a reference version.

    In fact, it has been. The last time I yelled at this developer for being a fuckup. So what's going on here? It uses an awkward, very inefficient reference implementation, which allows 2 very disparate modes of operation that, in practice, are rarely mixed, and has been known to perform like a dog (because it does twice as many joins and coalesces the two branches together in its results)

    So out goes the branch this thing doesn't use, and out go the coalesces. This also uses those values noted previously that are stored in the EAV antipattern. Just for funsies, I'll do the same fix here. It's not strictly necessary, since the row count here is so low, but whatever.

    Original execution time: 5 minutes.
    New execution time: 1 second.


  • Garbage Person

    And now, #2.

    SHOWPLAN permission denied in database 'tempdb'.

    Oh. This is going to be GOOD.


  • Garbage Person

    Alright. So what this thing is doing is pretty cute. It's generating XML in emulation of a feature on the newest version of the platform so it can use the same document layout as the Bill of Materials (BOM) from the new version. In order to do this, it bungs a bunch of values into temp tables and variables and then string concatenates a bunch of values, literals, and FOR XML output into a string, which it then returns as the single result in a single column.

    Seems I need to have a word with our project managers, too: You don't get new version features without getting the WHOLE new version. It's difficult to upgrade poorly designed applications because we intentionally took away all the pointy bits certain developers and PMs insist on impaling themselves on, and installed caution tape around the minefields.

    So some of the components in this BOM are apparently paper stocks. They need to be listed in a separate section. How are we figuring out which ones those are?

    LEFT JOIN (
       SELECT t2.id AS id, bomitems.id AS stockid 
       FROM #Temp2 t2
    	LEFT JOIN bomitems WITH (NOLOCK) ON t2.[ItemId] = bomitems.[id]
    	WHERE bomitems.Description like '%||LETTERSTOCK@$%'
    ) as stock
    

    Description is a human readable field. Allegedly.

    This procedure is stupid, but I'm having a tough time coming up with a better way to do it. I'll just dump the transaction isolation level down to READ UNCOMMITTED so it doesn't lock reference tables and be done with it.

    If it remains a problem, maybe I'll come back to it.


  • Garbage Person

    #1: Record count.

    What it does: Takes a filename as input. Does something utterly fucking incomprehensible. Seriously, I can't follow it. It calls the reference implementation of the "Get list of documents groups we need to output" procedure. I don't know why. I am the fucking overlord of the entire platform this thing uses and this is exclusively using platform-generic standard shared tables and components. And I can't figure out what the hell it's doing. Returns (somehow) the number of records.

    What I changed it to do: Count the fucking number of records and return it.
    Was: 3 minutes.
    Is now: Instantaneous.

    Now that I've fixed 3 of the 4 busted procedures, I'm going to adjust the script to use my new procedures and see what happens.


  • Discourse touched me in a no-no place

    @Weng said:

    see what happens.

    This is where all hell breaks loose.


  • Garbage Person

    Did I mention I'm doing this straight in production? (because rules only apply to mere mortals and we're operating in "missing SLAs by the minute" mode)


  • Discourse touched me in a no-no place

    I have to admit, I was kind of hoping you'd create a new account and never post from this one again. That would've been pretty epic.


  • Garbage Person

    I'm not done editing scripts yet.

    Finding more WTF's.

    Clear copypasta from other programs (it uses their unique identifiers). This, naturally, does nothing to this program but has a small outside chance of corrupting data for THOSE programs (this is fixed in latter versions of the platform - we don't require programmers to specify their program's unique ID anymore - we do it for them)

    Branches of logic that are 'live' in that they are technically reachable under certain circumstances but clearly have not been updated in years and will therefore do the wrong thing.

    SELECT 'false' AS false
    

    stored into a variable called truefalse.


  • Garbage Person

    Unfortunately, everything has gone well. Double-unfortunately, performance still unequivocably sucks balls. I'm going to let it run in prod now at a greatly throttled speed, head out to get some dinner and then return to tackle the BOM procedure.


  • Garbage Person

    Things this procedure does:

    1. Loads all the orders we're operating on into a temp table.
      Nearly instantaneous. No problems.
    2. Loads all the line items for those orders into a temp table.
      Nearly instantaneous. No problems.
    3. Indexes those temp tables.
      Seems excessive, but whatever. Instantaneous.
    4. SELECT DISTINCT @variable=some_value FROM <massive pile of joins, most of which are unnecessary, including several on subqueries>
      Takes quite a long fucking time to execute. (I didn't wait).
      Ah. Well then. That's a thing. @variable is a scalar.... DISTINCT is a bit redundant - if there's more than one value it's going to get the last one. And if there's more than one value in this context, it's broken. TOP 1 and nuke the unnecessary joins and subqueries.

    Total execution time is now 17 seconds, down from 3 minutes.

    Lets keep going through this and see what else I can shake out.


  • Garbage Person

    1. Sets a variable to a value that's that's present in the * of the query in #4, but via a different and frankly idiotic join path. I'll just set that variable up there.... (This one properly uses TOP 1)

    2. A query with several passes at the aforementioned EAV table. But it doesn't join to any other tables and only extracts data from that table, so all kosher. It does a DISTINCT, changed to TOP 1 (the values that it selects will only have 1 value per run of the procedure, but many rows). In it's WHERE clause, it has a function call - the result of the function is deterministic, and the only parameter is actually one of the sproc's parameters - so it's the same every time this is called. As clever as the query planner is, it'll execute that (fairly expensive - has a ton of joins) function for every row it's evaluating. So I'll kick that function out to a variable and run it once.

    3. Finally, actually goes and gets the detailed BOM data. Seems to be pretty well put together if overly complex.

    4. Loads of string manipulation to build the XML output.

    New total execution time: 1 second. Think I'm done here.


  • Garbage Person

    So the variable I set in #5. Turns out that in the original, it doesn't even fucking WORK. Removing it to be bug-compatible.....


  • Garbage Person

    So in summary: 4 hours of rockstar programmer (humble, ain't I?) time (including a 1hr dinner break and a lot of time spent live-blogging the session). Non-billable. Unpaid. On a Sunday. Maybe with a wee bit of comp time or some shit. So that's quadruple goodwill costs.

    All the things I was supposed to work on this weekend didn't get done (oh, yes, there were things I was SUPPOSED to be working on) and therefore get to wait until next month's maintenance window.

    The immediate fires are out insofar as I engaged in firefighting by the expedient of dropping fuel-air explosives on top of the inferno, but the whole thing is an ill-designed smoldering ruin now. It ain't fixed. Not even close.

    Nobody's going to get hanged. My demands and/or suggestions that someone be tasked to fix this piece of shit properly will be ignored (or they'll give it to the guy who fucked it up in the first place).

    I'm mopping up the last of the outstanding SLA casualties from Friday. Death toll is in the millions.


  • FoxDev

    you, my dear friend, need a new job.

    how do you feel about moving to Maine? my company has some openings for DBA/DBD that we'd love to fill, and a couple for Web Programmers/webserver adminsitrator.

    PM me if interested.



  • @Weng said:

    I don't think he knows what means

    he's not alone.


  • Garbage Person

    I'm only good at databases because I have to be. Nobody else is, and the DBA's sure as shit aren't going to help. I've learned it all at the pointy end. A good, educated DBD should be able to kick my ass. And once you get to the true DBA-level stuff, I'm basically taking my best guess and usually get lucky.

    And I hate the web with the burning passion of a thousand suns.

    I like what I allegedly do: Designing large, enterprisey systems; large, enterprisey workflows; and large, enterprisey processes. And then executing on those designs, all the way down to getting into the weeds and exact implementation details. It's dynamic. It's engaging. It's exciting. It's challenging. It's pretty well paid. I'm good at it. I get to have other, capable developers do the crappy parts.

    I don't like the extra crap I have to do because nobody else is capable of doing it in a way that it isn't going to politically obstruct my doing my real job.


  • Garbage Person

    Dammit. I was anonymizing and replaced it with <domain-specific, but very central-to-the-whole-damned-concept term> but forgot I had to manually HTML-encode.

    Fuck Dicsores.



  • You sound like me. Except that you at least have a DBA somewhere in the building.

    I have to wear 4 hats, and then then Corporate IT (on the other side of the continent and outsourced to hell and back) like to throw shitty MS Access 2007 "applications" at us and pretend that it's a workflow management system.


  • Garbage Person

    DBA? In the building? They aren't even in the same company. All our core IT functions (servers, admins, helpdesk and 'core boilerplate' type applications) are oursourced. And only the helpdesk is actually outsourced to an IT Services org. Servers, admins, and apps are outsourced to some random company whose CIO played golf with ours or something.

    I recently requested 5TB of disk space and received an estimate sufficient to buy and operate a triple-redundant 240TB SAN. And that estimate is per year.



  • Ooh, ooh! Do your IT also have a raging hardon for Citrix and insist that everything can be run through it and will be better that way, never mind that they don't know how to set it up so it at least integrates with the client environment, rather than confusing the shit out of the end users with missing file type allocations and virtual drives?


  • Garbage Person

    No. We have maybe two Citrix applications companywide.

    However, to make up for that sin, we don't have a Microsoft enterprise agreement. Every seat of every app is individually licensed at retail price. Often by individual business units with minimal centralized record keeping.

    Even after they audited us.



  • @Weng said:

    we don't have a Microsoft enterprise agreement. Every seat of every app is individually licensed at retail price.

    What would happen if your CTO were to find out that it's technically extremely easy to make MS operating systems and applications activate themselves using publicly available KMS volume license keys in conjunction with a fake KMS server?

    What would be the political result of your CTO acquiring this particular money-saving tip via the drone whose "work" you're constantly needing to clean up after?


  • Garbage Person

    Nil, because in an organization this size, money is no object (See: "I recently requested 5TB of disk space and received an estimate sufficient to buy and operate a triple-redundant 240TB SAN. And that estimate is per year.") and it's easier to just have IT drones the serial number from the sticker.


  • Discourse touched me in a no-no place

    @Weng said:

    in an organization this size, money is no object

    Umm, probably not. It's usually more “as long as you stay in budget, nobody cares” with a side helping of needing to spend the entire budget by the end of the relevant fiscal year. Large organization finance is WTFy in many ways, but this part is at least easy to understand; it's all driven by the need for bureaucratic certainty in the face of reality…


  • Garbage Person

    No. That's how much our outsource datacenter actually charges. Everyone acknowledges that it's impossible to budget for any sort of unexpected expansion when the prices are so out of proportion with reality.

    Seriously. SIX FIGURE annual quote for 5TB.



  • That will be the efficiency of private enterprise at work, or something.



  • @Weng said:

    @variable is a scalar.... DISTINCT is a bit redundant - if there's more than one value it's going to get the last one. And if there's more than one value in this context, it's broken. TOP 1 and nuke the unnecessary joins and subqueries.

    Actually, its better to write it as a subquery so that it errors if it returns more than one row. TOP 1 will just make the problem harder to find if you end up in a situation where there is redundant data.


  • Discourse touched me in a no-no place

    @Weng said:

    No. That's how much our outsource datacenter actually charges. Everyone acknowledges that it's impossible to budget for any sort of unexpected expansion when the prices are so out of proportion with reality.

    Seriously. SIX FIGURE annual quote for 5TB.

    I emphasise that I don't know this for sure, but is that with some sort of crazy level of availability, redundancy and backup? We used to have problems like you describe (our internal SAN was amazingly expensive) and found that most of the cost could be lopped off by opting for a lower level of service in a few critical areas (OK for what we were doing). For stuff like web server logs, hyper-redundant disks just aren't justifiable ever.


  • Garbage Person

    There are 3 tiers. Tier 1 has DR and backup. Tier 2 has backup. Tier 3 is JBOD.

    Production everything is required to be tier 1.

    Tier 2 is only about 10 percent cheaper.

    Oh, and the backups are run by agents installed on the servers, not at SAN level. I have a server that wastes 4 cores and 16 gigs of RAM 23 hours out of the day just compressing and encrypting it's own backups. CPU and RAM are also unobtainably expensive. Its a fucking racket.


  • Discourse touched me in a no-no place

    We used to have such things (might still have some vestiges left) but the budget holders rebelled and told IT that their prices were ridiculous and that they'd outsource services wholesale if IT didn't cut their prices and stop charging for shit the budget holders didn't need. IT backed down; we're a university and professors definitely hold the upper hand on most things financial.

    The reason for all the cost is probably that your service providers bought one of those amazingly expensive storage solutions (the ones that the likes of EMC sell) without considering properly whether it was the right solution for all cases. Someone needs High Availability, so everyone had better get it. Someone needs fancy backup, so we provision it out for all. Keep on taking the set-union of all of these requirements and you rapidly push the price out into the stratosphere. It's a mark of shame though: the people who authorised the purchase demonstrated that they didn't know the use cases (or were very good golfing buddies with the storage salesman).



  • Seems I need to have a word with our project managers, too: You don't get new version features without getting the WHOLE new version.

    Oh, how I wish I could get our PMs to agree to that. It's bad enough backporting the new version features to the older applications that can't be upgraded; I hate when they try to make me backport them to the ones they can and are just too lazy to keep up to date.


  • Garbage Person

    They don't have to agree, just obey


Log in to reply