Yahoo's Engineers Move to Coding Without a Net



  • The interwebz recently got me [url=http://spectrum.ieee.org/view-from-the-valley/computing/software/yahoos-engineers-move-to-coding-without-a-net]this article[/url] (go read it).

    Software engineers at Yahoo are no longer permitted to hand off their completed code to another team for cross checking. Instead, the code goes live as-is; if it has problems, it will fail and shut down systems, directly affecting Yahoo’s customers.

    “Doing that,” Rossiter told me, “caused a paradigm shift in how engineers thought about problems.”

    Apparently, Yahoo thinks continuous delivery and continuous integration can replace QA proper. Strange how it correlates with the simple fact that Yahoo is less and less relevant, and for me it's rather a signal that they are not working smarter but have cash flow problems and offset QA work onto developers – and of course, on unsuspecting customers.

    It has also, he said, forced engineers to develop tools to automate the kinds of checks previously handled by teams of humans. An engineer might go through an arduous process of checking code once—but then would start building tools to automate that process.

    Methinks if engineers didn't automate QA processes when there were humans readily available, this means the engineers had a shitty mindset, QA had shitty mindset, and/or management had shitty mindset. Maybe all of them. Three strikes, and you automate. But relying solely on what automated tests say puts you a powerful reality distortion field.

    I wonder, what could possibly go wrong with that in a longer run...

    Some started working on automation [for testing], and they thought that was great—that they didn’t have to do the same thing over and over.

    Oh, so some developers turned into makeshift QA. But the fact they just have discovered they can automate testing tells me they are quite shitty engineers... Good thing they improve, finally.

    I'm in QA now, and all I do is automate shit. There is another team dedicated to exploratory testing. Sometimes they catch really obscure things that can impact customers severely.

    Paging @Yamikuronue.



  • What happens when you take away the quality assurance team in a software development operation? Fewer, not more errors, along with a vastly quicker development cycle.

    Says who? Are there really less bugs or just less discovered bugs?

    I once heard of a certain forum software that became much more bug-free once its free QA team got banned...



  • I interpret this as follows: developers were lazy assholes that thought ALL of the testing has to be done by QA. Now that the company scraps at the bottom of the barrel, it cannot afford those, and now developers are forced to live in that hell, be fucking responsible, and automate their shit if they want, of course, to have less stress.

    That destructive laziness could be probably indirectly enforced by previous management action (some managers really like to stifle initiative, like, "you're supposed to code, and crank more code, leave the testing where it belongs!" And to QA: "What are you doing? Automating? You should be busy clicking around, we ain't got no time for that! Also, it's not on the budget.")



  • I also imply that for the engineers that remain the work turned into death race, combined with the fact they can no longer work from home.



  • Yes, of course, I'm not suggesting the methodology of just throwing things over the fence for QA to worry about is a good approach, I just don't think it's entirely responsible to can the whole QA team and pretend that this will improve the quality of the product.



  • Extremes, it's called.

    QA who cannot code are mostly obsolete, as are engineers who cannot do rudiments of QA. But I think a Jack of all trades is expert at none; I can do full stack web dev if I must, but I'm more comfortable at backend side, and my frontend will be invariably shitty. I can do testing, and I can try doing unexpected things to my software, but that kind of mindset is not really ingrained in me. I know too much of a code to think up a way of breaking it.



  • @wft said:

    I interpret this as follows: developers were lazy assholes that thought ALL of the testing has to be done by QA.

    I don't like testing and I would love if someone else did it for me.

    But the problem is worse than PHB asking me to test, usually they give me one or 2 days, when a tester would ask for a week or 2. They are fully aware that it will be a crappy test. In a critical system that causes customers to lose money when it fails.



  • You can have a luxury of writing your whitebox tests along with code. It's not an unreasonable thing to require.
    You, as a developer, should always have a nice set of developer tests. They are usually faster and if written well, help pinpoint the exact source of your problem.
    Also, writing a failing developer test for bugs your tests don't find but QA's do, is also a Good Thing.

    But if a PHB tells you to do end user tests, just clicking around in hopes something will pop up, just quit.



  • Having said all of the above, I'm wondering how do you, for example, unit-test a device driver.

    Currently trying to wrap my head around that for, hmmm, reasons.


  • Discourse touched me in a no-no place

    @Deadfast said:

    I once heard of a certain forum software that became much more bug-free once its free QA team got banned...

    More bug-free, or more discovered bug-free?

    😈

    That reminds me - we're a few updates behind...

    We may be missing new 'features.'



  • @Deadfast said:

    I just don't think it's entirely responsible to can the whole QA team and pretend that this will improve the quality of the product.

    I would expect just that simple move to improve the measured quality of the product. In much the same way, we could substantially wind back the autism epidemic tomorrow if we just got rid of all the paediatricians, and lower the obesity rate by eliminating scales.



  • @wft said:

    I know too much of a code to think up a way of breaking it

    Fuzz testing FTW!



  • @wft said:

    I'm wondering how do you, for example, unit-test a device driver.

    Hardware test harness, generally.



  • @flabdablet said:

    Fuzz testing FTW!
    So you do know how to test smart! Why not do it that, then? Or does your PHB insist on all-manual mode?



  • @PJH said:

    That reminds me - we're a few updates behind...

    We may be missing new 'features.'

    Yes, "features"...



  • @wft said:

    Having said all of the above, I'm wondering how do you, for example, unit-test a device driver.

    Currently trying to wrap my head around that for, hmmm, reasons.

    In my cracking times, we used softice for ring-0 debugging. I don't know what the kids today use


  • area_deu

    I like the sad smiley they use to tell you there are upgrades available. "Oh no, new bugs fresh from the bikeshed!"


  • Notification Spam Recipient

    @wft said:

    QA who cannot code are mostly obsolete
    Most of what you say rings true but I've seen the code that most QA write and I've ended up having to maintain/rewrite most of it. Please stop them writing code. QA should make my life a misery and I should try to make their lives easier by writing less shit code and automating as much of their stuff as possible.

    Developers that don't write tests are retarded and are just asking for trouble. They're dull and tedious and rarely an indicator of overall software quality but they help keep the wolves from the door.



  • A quote I heard from a QA guy I used to work with seems appropriate here

    [Quote]
    Developers are optimistic testers at best. You need QA to really break shit, otherwise your customers become QA and then you have no customers
    [/quote]



  • @Nocha said:

    You need QA to really break shit, otherwise your customers become QA and then you have no customers

    something something Discourse something



  • @DogsB said:

    Most of what you say rings true but I've seen the code that most QA write and I've ended up having to maintain/rewrite most of it.

    You say this as if your code never needed rewrites. Or as if you turned competent by magic, overnight.

    QA should be specialized developers, not clickers-turned-coders-in-three-weeks. Responsible coding has a learning curve, and you can hack it only to limited extend; otherwise it needs its time to sink in, lots of throwaway work, and lots of peer-reviewed code. There's no way around that. If you have responsibilities as mind-numbing as a manual regression suite which takes X of your working time, learning abilities are offset, and not by X, but by (1.5...2)X.

    But my question stands: how the fuck are you supposed to get good code out of them, and generally expect them to work smarter, if you don't let them code?

    There are programmers specializing in databases, servers, UI/UX, systems, and there are programmers who can test the living shit out of other programmers' code. The latter are/should be the QA people in IT projects. Plus maybe a few who are top notch in the actual problem domain.


  • Notification Spam Recipient

    @wft said:

    You say this as if your code never needed rewrites
    I've no idea where you pulled that out of but if it were up to me my code would be deleted and I would fuck off into the sunset never to work again but seeing as I need a pay cheque and for some reason I haven't been fired yet that isn't going to happen.

    @wft said:

    QA should be specialized developers
    No they shouldn't be. QA should be breaking the product and doing whatever else it is they do to make the product better. Not having their minds broken by the tedium of automating tests.

    @wft said:

    not clickers-turned-coders-in-three-weeks
    That is changing but it's still the norm. YMMV. You're seeing more and more positions for "Developers In Test" but for the most part what you see is graduates/interns given that crud work or QA sent on three day courses and expected to turn out automate tests in their spare time which there is none of because they're been redeployed to different branches once they're current workload is exhausted.
    Also good luck keeping "Developers In Test" for any period of time. QA is fucking tedious and hard. There's a reason why good QA are worth their weight in gold and are usually paid about that.

    @wft said:

    But my question stands: how the fuck are you supposed to get good code out of them, and generally expect them to work smarter, if you don't let them code?
    I don't want them coding. It rots their minds and in general leads to them having sympathy towards a developer where none is warranted. You pay them to break shit and keep the wolves from the door. Is it glamorous no. They're just like developers. Another fucking cog.

    @wft said:

    There are programmers specializing in databases, servers, UI/UX, systems, and there are programmers who can test the living shit out of other programmers' code. The latter are/should be the QA people in IT projects.
    They're welcome but they shouldn't be the only level of QA. QA and Programming are very different skills with different mindsets. If you have someone who has married the two. Wonderful. In my experience people tend to pick a role and stick with it. Usually QA who dabble in programming either give it up or move onto a place where they can take it up full time.

    @wft said:

    Plus maybe a few who are top notch in the actual problem domain.
    Worth their weight in gold.



  • @DogsB said:

    Not having their minds broken by the tedium of automating tests.

    Instead, you prefer a mind broken by the tedium of manual regression cycles?
    In most settings I've seen, actual exploratory testing meant to break shit is, like, 1% of all testing, if not less.

    Automate what you know and what you knew broke shit previously. Write bulk data generators. Use fuzzers and whatnot. There is a plenty of techniques of testing that involves the machine sweating over it, saving the mind for more creative ways to break shit.



  • I suppose this explains why their webmail client has progressively grown slower and buggier over the past couple years.


    Filed Under: What, you expect the options bar to become enabled when you select an email? DOING IT WRONG!


  • Trolleybus Mechanic

    So rather than spending X hours coding, and having someone spend Y hours coding (where the hourly salary for Y is << than X), you have someone spending 2X hours coding and testing. This means you have far less software being developed, because the coders are spending half their time doing QA.

    Wait, sorry, my mistake, I accidentally tried to insert reality there. Allow me to translate to CEO Language (aka: Functional Sociopath Speak)

    "If I fire the whole QA staff, total expenditure goes down. On the books, that is money saved. There is no quantified way of measuring code delivery in the short term. So for this quarter, profits will go up. I will get my bonus. By the time the smouldering fire bursts into an inferno of burnt-out coders, broken software and missed deadlines, I'll be on to my next position with a fat cashed check! I'm the best!"



  • @Nocha said:

    A quote I heard from a QA guy I used to work with seems appropriate here

    [Quote]
    Developers are optimistic testers at best. You need QA to really break shit, otherwise your customers become QA and then you have no customers


    [/quote]

    This may be a WTF, but...

    I work on an internal application that has no QA team. Our "customers" double as our testers. And no, they don't do this testing on Production... we prefer that they break a server that's using a slightly out of date copy of our DB rather than break the production data. ...not that production's data hasn't had issues anyway. Heck, we've even discovered bugs in Oracle Spatial that Oracle had to fix.


  • ♿ (Parody)

    This reminds me, to an extent, of changes that happened in manufacturing. Originally, they'd just sample or inspect at the end and deal with that. Then they realized that the sooner they caught that the better off they were, so the people on the line became responsible for quality, not just the QC department at the end. Not that you still don't get the checks on the back end.

    But then, ensuring quality in software is almost nothing like mass production. I mean...style guides and static analysis, where available can do some things for you, but there's a lot of imagination and independent thought required to really test software.



  • That sounds like a form of User Acceptance Testing. The important part being that you don't deploy to production until after anything they have found is fixed. Imagine handing it into production before they have had a chance to do their bit...


  • Fake News

    @powerlord said:

    Heck, we've even discovered bugs in Oracle Spatial that Oracle had to fix.

    1. It's Oracle.

    2. It's "customer QA" all the way down.


    Filed under: Captain Obvious away!



  • @wft said:

    “Doing that,” Rossiter told me, “caused a paradigm shift in how engineers thought about problems.”

    Usually resulting in a developer shift, where all the engineers who can shift into another company, do so.

    @Deadfast said:

    What happens when you take away the quality assurance team in a software development operation? Fewer, not more errors...

    less discovered bugs

    This was my immediate instinct as well.

    @wft said:

    some managers really like to stifle initiative

    I've worked in some sufficiently toxic environments that I didn't know whether I should be automating or not, even though in hindsight I totally should. It wrecks your ability to think. Management are playing the blame-game while you try to work, and you get drawn into a way of thinking along the lines of "which way can I justify as most likely to be successful in the first place", rather than just automating the thing you spend about 15 minutes of every morning doing.

    @Lorne_Kates said:

    "By the time the smouldering fire bursts into an inferno of burnt-out coders, broken software and missed deadlines, I'll be on to my next position with a fat cashed check! I'm the best!"

    Learning when to bug out is an important skill. I'm well into my career and I'm just picking it up. My ex-manager actually tried to threaten me over this, of course, but that merely justified my case for leaving.



  • @wft said:

    Having said all of the above, I'm wondering how do you, for example, unit-test a device driver.

    You could create a simulator of the device, feed it to a virtual machine and on the host run something that will control the simulation and observe whether the virtual machine responds correctly.

    Of course you should still have the hardware rig, and automate it as much as possible, but that is automated integration test and not unit test (even the previous is rather dark box test for true unit test, but some things are not really unit-testable much).



  • This post is deleted!


  • @mott555 said:

    I suppose this explains why their webmail client has progressively grown slower and buggier over the past couple years.

    Yahoo[i]![/i]'s internal[i]![/i] bug[i]![/i] tracker[i]![/i] (maintained[i]![/i] by[i]![/i] the[i]![/i] QA[i]![/i] testers[i]![/i]) shows[i]![/i] no[i]![/i] new[i]![/i] bugs[i]![/i] since[i]![/i] the[i]![/i] first[i]![/i] quarter[i]![/i] of[i]![/i] 2015[i]![/i].


  • I survived the hour long Uno hand

    @wft said:

    Paging @Yamikuronue.

    Present!

    @wft said:

    Yahoo thinks continuous delivery and continuous integration can replace QA proper.

    Nah, they already replaced QA; the way I read the article, they were swapping code with other devs for an impromptu sort of code review.

    @wft said:

    QA should be specialized developers, not clickers-turned-coders-in-three-weeks.

    Quote for FUCKING truth. The guy I've got now is a junior level guy, but he has a development mindset, and that makes him worth way more than the "experienced" clickers I've had in the past.

    @DogsB said:

    Not having their minds broken by the tedium of automating tests.

    Fuck you. You know what's tedious? Regression testing. "Yay I can sort products, just like I could last week." Automating the hell out of that shit is way more interesting and engaging than doing it time after time.


  • Notification Spam Recipient

    @wft said:

    Instead, you prefer a mind broken by the tedium of manual regression cycles?
    That's why it's in the best interest of devs to automate that shit.
    I honestly don't think there is a good outcome here. Good QA will always be required everywhere. You want to automate as much as their work as possible but dev doesn't want to write/maintain those tests. QA hate the soul destroying regressions but there isn't enough time to get them to get them up to a level where the code won't make your eyes bleed and have to rewrite it or worse watch them go ignored because no one can decipher the test.

    @Yamikuronue said:

    Fuck you.
    I charge hourly for a minimum of two hours. Finishing before the end of an hour incurs a charge for the entire hour. There is a by the minute option if you wish but you will incur travel fees then.

    @Yamikuronue said:

    You know what's tedious? Regression testing. "Yay I can sort products, just like I could last week." Automating the hell out of that shit is way more interesting and engaging than doing it time after time.
    Then produce a cost benefit analysis for your project manager including plans for on-going maintenance.


  • I survived the hour long Uno hand

    @DogsB said:

    Then produce a cost benefit analysis for your project manager including plans for on-going maintenance.

    I have. You get far better ROI for me and my contractor maintaining an automated regression suite than for paying six idiots in India to manually test.


  • Notification Spam Recipient

    @Yamikuronue said:

    @DogsB said:
    Then produce a cost benefit analysis for your project manager including plans for on-going maintenance.

    I have. You get far better ROI for me and my contractor maintaining an automated regression suite than for paying six idiots in India to manually test.

    This is where I got my start actually. Automating regression tests. The problem was getting people to maintain them which I took up for my group when I was made permanent. It was hard work but we had the lowest regression rate and found defects faster than anyone. Early mornings were usually me roaming the office looking for blood. The problem is that management hire contractors/badger testers into writing them. Dump them on someone else to maintain them. Watch them go un-maintained and watch the bugs come back because the testers aren't covering that ground any more.

    I have no doubt that your ROI could turn into a daily regression run which will catch a lot more a lot faster. The problem I have is who maintains them but you have that sown up. :)


  • I survived the hour long Uno hand

    @DogsB said:

    who maintains them?

    The two full-time QA people that my company seem to think are sufficient for a dev team of almost two dozen: myself and my contractor. TRWTF is my company refusing to put together a real QA team, but that's a whole nother story. But yeah, that's the biggest expenditure.



  • Well, Yahoo was irrelevant before. This is part of them trying different things to regain relevance (and cut costs, I'm sure).

    Users really are the best QAers. On the other hand, most users don't want to be treated as QA.



  • I don't mind being treated as QA as long as I get paid for doing so. I really object when I am paying for the privilege of being QA for somebody. Even more so if I can't use it as work experience and get a job out of it...



  • They tell the best team size is four, in terms of agility.



  • No wonder Yahoo wants to spin off Yahoo and become a holding company for it's Alibaba shares. I bet those guys are a bit more cautious with their code because, you know, they have users.



  • @Lorne_Kates said:

    "If I fire the whole QA staff, total spend goes down."

    BTFY



  • I practice bug-driven development. All pull requests must add at least one bug.


  • BINNED

    What if I request a bug, do I get a feature with it for free?



  • @Onyx said:

    What if I request a bug, do I get a feature with it for free?

    Probably can "request one get one free".



  • Sequel to TDD and BDD: DDD (Defect Driven Development)


  • BINNED

    Isn't that what Discourse is doing? Maybe they can invent a cool new name for it? 4D? 3D Development?



  • Yes, and there have been rumors that my company will take a similar approach to the Yahoo/Discourse method in our new ____ ____ initiative, but we will hopefully avoid that.


  • ♿ (Parody)

    @Onyx said:

    Isn't that what Discourse is doing?

    They claim CDD, but really it seems more like DDC.


Log in to reply