Open source testing (yes, another SMF story)



  • So, last year I wrote an upgrade to one of SMF's features. Anti-spam Q&A support was already a thing but it only allowed one answer per question and questions were ignoring languages (which meant it was only questionably useful on a multi-language site)

    And I added a new thing to fix this. But there were calls for it to be available for the current version given the timescale before the new version is ready, so I dug out the code and prepared it as an add-on for the current version.

    I say add-on, but I really mean 'describing in XML what bits to replace with what', but since the code is basically a one for one transition, that's not a huge problem. And then I submitted it to their add-on site.

    All add-ons get reviewed by the team prior to being publicly available. And mildly surprisingly, they found a bug. If a language had no questions available (but other installed languages do), it would just fail with an error. My bad, doesn't fail gracefully.

    What I want to know is how come they can find this issue in 24 hours when it's packaged as a mod but no bugger tested it in the last 10 months otherwise? I suggested I should backport all new features as add-ons, might get more test coverage that way.



  • Maybe they had automated tests that only run from their XML tester. Assuming they have one.

    In my experience, forum software tends to not be well tested.



  • No, no they don't. I've been on the other side of the curtain. I know exactly what testing procedures they have (nothing implemented, it's all through-the-interface testing, and frankly I'm amazed they even did that)



  • @powerlord said:

    In my experience, forum software tends to not be well tested.

    Even Discourse?



  • @Keith said:

    Even Discourse?

    Sorry. my sarcasm detector is broken, but I'm just going to assume you're being sarcastic.



  • @powerlord said:

    Sorry. my sarcasm detector is broken, but I'm just going to assume you're being sarcastic.

    A fairly safe assumption.



  • Thing is, I wouldn't disagree with you. SMF isn't well tested either in my experience. I keep finding commits from their lead developer that have never been tested in any fashion prior to commit.


  • Discourse touched me in a no-no place

    Come work with me - it'll make you think your testing is good.
    Unfortunately most if not all of the worthy WTFs can't be posted without anonymising them so much they lose the WTFiness.


  • Discourse touched me in a no-no place

    @Arantor said:

    What I want to know is how come they can find this issue in 24 hours when it's packaged as a mod but no bugger tested it in the last 10 months otherwise? I suggested I should backport all new features as add-ons, might get more test coverage that way.

    Why wasn't it found earlier? Because nobody really looked, or even perhaps the right person didn't look. Shit happens like that.

    You can't backport everything — anything that needs major code surgery isn't going to be happy that way — but it's often a good idea to make features available as add-ons so that they can get wider testing than you can do on your own. One of the best pieces of code I ever wrote started out like that. The only thing to be aware of is getting stuck supporting the old code for ages despite it being a transitional entity; you've got to manage user expectations carefully.



  • It's because no-one looked. Like most of the stuff in the upcoming version, shit's broken because stuff has been around for 1-2 years without being tested even by the people who wrote it. When I was part of the dev team, I was finding new bugs literally every few days, despite being assured that the in-dev version was 'basically ready to go', which it really wasn't.

    The thing is, yes, I can't backport everything. But the point is, they're busy arguing about stupid shit instead of doing any testing and one of the developers said yesterday that they'd rather not have any bug reports during this alpha phase.


  • BINNED

    @Arantor said:

    they'd rather not have any bug reports

    Just tell them there is a easy solution for this: don't put any bugs in their or get them out during you own damn tests before anybody else sees it and files a bug report. And maybe bring out the cluebat.
    I thought the hole idea of alpha/beta/gold was that one stops putting new shit in and so everybody can focus on making the whole thing work.



  • Well, they're still in the alpha stage but because the codebase is a ball of mud, unit testing is near enough impossible, and I was suggesting - with futility, I know - that doing incremental testing might go some way to remedying the problem. There's chunks of code that don't work, have never worked since they were written because they were never even tested by their authors before check-in, and have been lain around dormant for anything up to 2 years.

    Even doing some testing would limit the scope of this crap.

    See, even in alpha, some testing should be done, even if it's only proving that shiny new feature you just implemented actually works.


  • Discourse touched me in a no-no place

    @Arantor said:

    See, even in alpha, some testing should be done, even if it's only proving that shiny new feature you just implemented actually works.

    Propose that they do a big show-and-tell to people so that everyone can find out about all the wonderful new stuff. Live video stream or in person. Awesome way to show off the New Sparklies! (And to test shit at least a bit. Hey, it's all in how you market the testing…)



  • See, I actually did that. I started a trend of monthly blog posts showing off what was going on. No video because I'm a sociophobe and dislike the sound of my own voice, but pictures certainly.


  • I survived the hour long Uno hand

    I have a meeting today about why we should be using test doubles/mocks for unit testing.
    This is a hot-button, contentious issue.
    This is the third meeting on the topic.
    My boss promises that today we'll really truly not take "that's too much work" for an answer.....


  • Discourse touched me in a no-no place

    @Arantor said:

    See, I actually did that. I started a trend of monthly blog posts showing off what was going on. No video because I'm a sociophobe and dislike the sound of my own voice, but pictures certainly.

    You need to persuade someone else to do videos or presentations. You won't win on this if you insist on doing it all yourself. I know, it's faster if you just get on with it, but that's not the point. The real point — the one you're not telling to others — is to make someone else realise what a deep pile of shit you're all in and start evangelising for getting it fixed.

    Don't be the one guy who is a stuck record. Learn to manoeuvre others into being your sock puppets. 😉


Log in to reply