Programmer turned tester keeps seeing things, terrible things



  • (By the way, QA is a thing. Why won't NodeBB recognize it as a tag?)

    A gem from my work I've hit recently.

    You know why the project managers keep complaining that the QA budget is over-inflated, and some even get those crazy ideas that only unit tests should be allowed? Or, that end-to-end testing is only to be conducted manually? The truth is, well, because the QA budget is over the roof. And over the roof it is because developers seem to try really hard to make the damn thing difficult to test.

    A project I'm working on, wearing a QA hat, is a single application page. It pokes at the API here and there, and sometimes it just so happens that we need to make sure an API operation has completed before we go to the next step. First, the testers inserted waits generous enough for the operations to complete, but there were a few problems with that, the most obvious being that the tests were running dog slow. And then, some operations still did not finish on time even after those waits, and when the testing script chugged along, the state of the whole thing got really funny to debug. It was really hard to say where exactly was the point of failure.

    So the testers, after quite a period of taking hits from the management, did the right thing and complained to developers. They told they needed something on the page to tell them it's safe to proceed. The developers scratched their heads, and thought up a solution.

    There is a little library that shows cute, unobtrusive pop-ups, and hides them after a while. It can tell that something succeeded, or that something failed. Wow, said testers, and went happily on with their business. Wow, said developers. This is killing two birds with one stone! You know, end users also appreciate feedback!

    It quickly turned out that the cute pop-ups are not without their gotchas. First, we test them with Selenium. A Selenium web driver is a thing that you talk to with a kind of HTTP client. And the driver uses some kind of IPC to talk to the actual browser. What you get is actually so racy that there were times that scheduling the web driver to wait for the pop-up happened only after the pop-up came and went. The nasty thing about that pop-up is that it's not lurking there hidden, it creates the DOM elements when it needs to show something, and destroys them after expiry. The testers, in their ineffable wisdom, wrote the routine that checks for that pop-up and its contents, in two parts, each taking a roundtrip through the webdriver-actual browser chain. What could possibly go wrong?

    Thus, we've now got tests that pass only a part of the time.

    I think that fighting and patching those pop-ups has eaten several man-months already.

    For some strange reasons, I begin to sympathize with PHBs.


  • I survived the hour long Uno hand

    @wft said:

    They told they needed something on the page to tell them it's safe to proceed.

    This is the Right Answer, but almost never ends well. Poke me tomorrow and I'll share a handy code snippet I use in my tests instead.

    The reason it's the Right Answer, by the way, is that each AJAX request should have a purpose. When something happens, it should change something about the page the user can observe. Otherwise, why are you making calls at all? This doesn't take in to account pre-caching items after the visible page is loaded, however, or other nifty tricks developers like to pull off.


  • FoxDev

    @wft said:

    By the way, QA is a thing. Why won't NodeBB recognize it as a tag?

    Tags have to be at least three characters


  • FoxDev

    @RaceProUK #QA_?



  • @RaceProUK Then tags should be allowed to consist of 2 characters, too. There is a metric crapton of 2-letters abbreviations...


  • Notification Spam Recipient

    @RaceProUK

    @RaceProUK said:

    Tags have to be at least three characters

    So, QA_? 👅

    Edit: :hanzo: 'd.


  • FoxDev

    Sure, I'll use the admin creds I don't have to get that sorted right away



  • What does your application do?



  • @wft Alternatively, just tell your devs to introduce a setting in web server config that, when seeing it, will not automatically dismiss the dialog. Instead, your testing routine will be responsible to call the corresponding callback that does the remaining tasks when the dialog is about to be close and then close the dialog.

    That's how we bridge the test scripts up.

    Of course, it also mean you'll have to add a set of test for testing the callbacks are actually called when the flag is disabled. At least in those tests you don't need to check the values, just need to check whether the UI is updated.


  • area_can

    Oof, I've had some tough times with Selenium jankyness. It's absolutely terrifying when your tests sometimes pass, or when it turns out that someone changed the delay time for some UI component and that ended up causing a dozen tests to fail in extremely weird ways.



  • @Yamikuronue said:

    Otherwise, why are you making calls at all?

    Analytics, refreshing session/auth tokens, polling status of some long-running thing, etc. Calls that don't necessarily result in UI updates do exist.


  • I survived the hour long Uno hand

    @CatPlusPlus True. But none of that should be blocking UI or you're officially :doing_it_wrong:


  • area_pol

    @wft said:

    First, the testers inserted waits generous enough for the operations to complete

    Let us imagine that the user performs the next step very quickly. The application should not crash.
    If it crashes, the test is failed.
    Otherwise, it should tell you that you performed the action before the backend was ready - by keeping the button disabled, or by showing an error message if you proceed before it is ready.
    So the automatic test could click it every 0.05s and wait until application says it is ready and accepts.

    If the whole indication is a popup, it will be ignored by the users. You do not want to be an author of software with arcane rules "don't click this until you see a popup because it will crash".



  • @Yamikuronue said:

    blocking UI

    IO should never ever block UI



  • @ben_lubar said:

    IO should never ever block UI

    Ever hear of these things called "command-line applications?" :trollface:



  • @Groaner even then, if I'm interacting with the CLI in any way, IO shouldn't block. For example, I can hit ^C on less and it'll stop waiting for disk, which wouldn't be possible if it ran IO on the UI thread.



  • @ben_lubar said:

    For example, I can hit ^C on less and it'll stop waiting for disk, which wouldn't be possible if it ran IO on the UI thread.

    Actually it would be, signals are interrupts: the process wakes up and drops whatever it's executing and switches to the signal handler (which is why you gotta be real careful with what you're doing in signal handlers). less in particular has more complicated UI so maybe it does have a separate thread for that, but in general I/O can be interrupted just fine even on a single thread.



  • @Yamikuronue said:

    @wft said:

    They told they needed something on the page to tell them it's safe to proceed.

    The reason it's the Right Answer, by the way, is that each AJAX request should have a purpose. When something happens, it should change something about the page the user can observe.

    For example, when it comes to changing something on a grid and saving it to the database, there's no visible difference. The UX team didn't think of differentiating saved/unsaved records.



  • @CatPlusPlus said:

    @ben_lubar said:

    For example, I can hit ^C on less and it'll stop waiting for disk, which wouldn't be possible if it ran IO on the UI thread.

    Actually it would be, signals are interrupts: the process wakes up and drops whatever it's executing and switches to the signal handler (which is why you gotta be real careful with what you're doing in signal handlers). less in particular has more complicated UI so maybe it does have a separate thread for that, but in general I/O can be interrupted just fine even on a single thread.

    It doesn't have separate thread for terminal control, otherwise arrow keys would be usable while it loads a huge file.


  • I survived the hour long Uno hand

    Here's the snippet I promised you. It requires jQuery, which just about everyone's using these days, and 0 changes from the devs[1]:

    /**
    	 * Wait for AJAX to complete. Requires jQuery. This will return when jQuery.active is false.
    	 * @param driver The driver to hold up
    	 */
    	public static void WaitForAjax(final WebDriver driver)
    	{
    		 Wait<WebDriver> wait = new FluentWait<WebDriver>( driver )
    				    .withTimeout(120, TimeUnit.SECONDS);
    				 
    				 
    		  wait.until( new ExpectedCondition<Boolean>() { 
    		    public Boolean apply( WebDriver webDriver ) {
    		    	return (Boolean) ((JavascriptExecutor) driver).executeScript("return jQuery.active == 0");
    		    }
    		  } );
    	
    	}
    

    Another useful snippet if you're dealing with fiddly controls and JS hooks that sometimes just don't run when you click:

    /**
    	 * Click using Javascript. Use whenever clicking fails reliably. 
    	 * @param element The element to click
    	 * @param driver The driver to use to do the clicking.
    	 */
    	public static void clickUsingJavascript(WebElement element, WebDriver driver) {
    		JavascriptExecutor executor = (JavascriptExecutor)driver;
    		executor.executeScript("arguments[0].click();", element);
    	}
    

    And one for scrolling the viewport so you can interact with stupid controls that need to be on screen to render properly:

    /**
    	 * Scroll the viewpoint to make a certain element visible. This should happen automatically before clicking, but sometimes it can be fiddly. 
    	 * @param driver The webdriver to scroll
    	 * @param element The element to make visible.
    	 */
    	public static void ScrollElementIntoView(WebDriver driver, WebElement element) {
    		((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", element);
    	}
    

    1: If you're doing test automation, you're still doing programming and thus a programmer, you're just not a developer anymore


  • kills Dumbledore

    @Yamikuronue I've recently taken over the Selenium based testing suite for a bunch of websites we run. These could come in very handy, thanks


  • I survived the hour long Uno hand

    @Jaloopa Yeah, NodeBB notificed me that you bookmarked it O.o


  • kills Dumbledore

    @Yamikuronue Legitimately, not for trolling porpoises (for once)


  • BINNED

    @Jaloopa said:

    trolling porpoises

    bookmark trolling!


  • BINNED

    @thegoryone said:

    @Luhmann This sounds like something I could get in on

    It was only guessing who would be frist to jump on the trollybus



  • @Yamikuronue Whoa, code sure looks nice here. No idea if it's just the slate theme.

    0_1458573671108_upload-0aa90a17-b7bd-42ed-8dbd-e341accd7859


  • I survived the hour long Uno hand

    @Magus I see the same in Superhero theme. Very nice!



  • @Yamikuronue Thanks, some of those snippets can be useful. We, however, are using Angular which wraps XHR in its own thing, so jQuery.active is not as handy anymore... (and in tests, we actually use NodeJS/Protractor, so the client is asynchronous too, and is "fun" in its own right).


  • I survived the hour long Uno hand

    @wft said:

    we actually use NodeJS/Protractor

    I'm sorry.

    (Webdriver on NodeJS is fucking atrocious compared to Java)



  • More new, exciting stuff. Or old, unexciting stuff.

    Our product uses jQuery UI widgets a lot. That is, they use the Grid from there. The grid, it turns out, goes bananas with regards to DOM and micro-optimizes the hell out of it, so it dynamically shoves the visible cells into it, and removes those that are not visible.

    I don't know how much time/memory/CPU it really saves, not my problem really, but the complexity of asking the browser "what is there in the column X" grew rather a lot.

    Previously, we could easily query for contents of the off-screen elements. Now, they are offscreen, they are off DOM. To get the ability back, we either need to use Javascript APIs the grid provides, or scroll around like mad to make shit visible first... Some team chose the second way. It kinda stuck. The tests fail sometimes.

    Well, to use the Javascript APIs, one also need to add a new webpack configuration and also make sure the CI environment uses it too. I think I will be the one doing that. As well as unfuck all the APIs and refactoring all the tests.

    It's a bit of a pity "chief unfucker" is not a valid job title.



  • @wft said in Programmer turned tester keeps seeing things, terrible things:

    We, however, are using Angular which wraps XHR in its own thing, so jQuery.active is not as handy anymore...

    You can use interceptors to get access to whatever the hell $http might be doing. Someone used that to make a YouTube-esque progress bar:


Log in to reply