"Unit testing" reserved for automation?



  • I want a sanity check to make sure I'm right on this one.

    Where I work, people keep calling testing individual products unit testing. So a whole set of pages that make a webapp. Or a functionality.

    But it's always testing it from the external UI. We never develop a test that we execute on a functionality.

    Every time I hear unit testing, I'm thinking automated testing. And I'm in an environment where that is not even on the radar.

    So, I get a little frustrated when I am told to go "unit test" something.

    I mean, at the point where you are testing the UI, even if it isn't the whole system, but just a sub-product. It's acceptance testing in my mind.

    Am I wrong here?

    Is this an acceptable use of the term, or not?

    I've resolved to replace the term with something else when I hear it. Module testing?



  • I'm with you on this, but hey, lately I've seen HR departments doing standups.



  • I can understand where the confusion comes from on non tech people that "I hit save, is the file there?" counting as a unit test of saving. If this is the way that the tech people are using it then it could be that they don't want to confuse the people that do requests for features and scheduling and so on, but yeah it isn't what is meant in normal groups by unit testing.



  • I thought that "unit testing" involves a moderate amount of cash, a transaction with said cash, and about 5 mins



  • I've heard this called unit testing by people who know better to make the project sound better. I was fairly confident of the quality anyway, or I would have probably posted it here.



  • I don't think standups are a part of the agile creed.
    So it's a byproduct of implementing the creed.
    So it can be used for any team that wishes to be accountable on a daily basis.

    I think I'm more offended at the fact that we spend so much time doing this old-school "acceptance testing" and calling it unit testing.

    The reason I'm offended is because if people don't share the real definitions of unit test, interface test, acceptance test, they'll never get serious about implementing actual unit tests.

    But it gets down to having to perform the same steps either way because we have to test the UI, so why automate testing anyway?

    Maybe I'll have to implement some kind of macro based UI testing.
    I have my project manager trying to talk about a QA position, but it's more about developing good testing practices, and unifying the formulation of tests for all deliveries of like deliverables.


  • kills Dumbledore

    Implement Uint testing - like any other test but forgets to check functionality if any number is negative



  • Do you work for the $vendor I deal with? Because they pull this exact same stunt, and I hate it.

    Also -- I'd call this 'bench testing', considering that this isn't a true unit test because it can't operate at the class or function level, and it's not an acceptance test either as those are performed on a finished release, not a developer's work-in-progress.



  • So I got confirmation from my product manager that he really does mean for me to automate testing.

    It's not unit testing, because that's a development test, but we definitely are heading in the right direction.

    Our delivery testing lasts way too long, requires way too much travel, and often times leaves us stranded with little support and capability to correct issues if they show up on site.

    My goal is to not just automate the test, but to identify and offer solutions as a troubleshooting system as well.
    A diagnostics / automated acceptance testing of sorts.

    I've never seen a UI failure. It's always been a configuration failure. So there's very little reason for us to confirm through a UI the results of our tests.

    This should be interesting.


  • ♿ (Parody)

    We have a suite of automated tests. They're really integration tests, but we generally refer to them as unit tests.


  • I survived the hour long Uno hand

    Unit testing is testing the smallest possible discrete unit. This is, in practical terms, just about impossible to do without a harness of some kind, and then you're halfway to automation anyways, so there's no real point in executing it manually, though you could rig up some kind of harness that takes manual input and spits the output onto the screen.

    There are two places I've seen manual unit testing done:

    1. In a service library, by having a unit test framework execute the method invoked to do the work of the service and then print the results to the screen for manual verification. This is stupid.

    2. In SQL, by listing a series of test cases for a stored procedure in the comments above the procedure definition so they can be executed manually to verify the cases. This is people trying to work around a lack of tools, and as such, more acceptable.

    Anything involving multiple components is not unit testing anymore.


  • Java Dev

    we've got a custom string find/replace language, which features a command line tool for dev use only. That's never really been used for manual unit testing though, although a couple years back I wrote an automated unit test with it (all our unit tests are bash scripts).



  • No it is not an acceptable term because it is incorrect.



  • @Yamikuronue said:

    2) In SQL, by listing a series of test cases for a stored procedure in the comments above the procedure definition so they can be executed manually to verify the cases. This is people trying to work around a lack of tools, and as such, more acceptable.

    More specifically, pgTAP. I don't see anything for other RDBMs though.



  • You don't see much for databases because DBMSs are more difficult to create automated unit test for because they tightly bind the method to the data.

    I'm not talking about CRUD tests, if you suspect the database isn't storing your data properly then you either need a better database platform or a psychiatrist.

    The problem with testing stored procedures is that you can't simply mock up the data storage and run the procedure, you have to actually store data, then run the procedure. This means you either have to run the procedure on a database that already has data in it - adding a dependency on that data, or create the data as a step in the test. Procedures that are date dependent (like a hypothetical GetOrdersForCurrentMonth) become a challenge. Procedures that require mutually exclusive data (testing the output of a procedure and testing the security in that procedure in the same test suite) are an even bigger challenge.

    There is also the challenge of verifying that you are actually testing the right code. Pretty much the only way to do this is the have the test suite start with an empty database and pull the schema definitions from source control to create everything from scratch.

    In other words - tools aren't the problem. The lack of tools simply represents the reality that testing at the database level is often more work than it's worth.



  • "I can haz unit test for .NET's IL add method?"
    "But that neez data on stack!"
    "Step 1: Put data."
    "Dependz on data! Too much work!!!"



  • @Jaime said:

    The problem with testing stored procedures is that you can't simply mock up the data storage and run the procedure, you have to actually store data, then run the procedure. This means you either have to run the procedure on a database that already has data in it - adding a dependency on that data, or create the data as a step in the test.

    Isn't this what the setUp() and tearDown() type hooks in an xUnit are for? You create a test double schema from a pre-populated 'master' copy, then simply drop the test double's tables when you are done.

    @Jaime said:

    Procedures that require mutually exclusive data (testing the output of a procedure and testing the security in that procedure in the same test suite) are an even bigger challenge.

    With my approach it shouldn't be, as it'll give you a clean slate for each test case.

    @Jaime said:

    Procedures that are date dependent (like a hypothetical GetOrdersForCurrentMonth) become a challenge.

    This is the same problem that is encountered by any code that depends on the current date.

    @Jaime said:

    There is also the challenge of verifying that you are actually testing the right code. Pretty much the only way to do this is the have the test suite start with an empty database and pull the schema definitions from source control to create everything from scratch.

    How do you make sure your test data is synchronized to your schema, then?



  • Actually, I'm pretty sure that there IS a CLR compiler for LOLCODE.


  • FoxDev

    @Magus said:

    Actually, I'm pretty sure that there IS a CLR compiler for LOLCODE.

    why does that not surprise me?



  • I still think it would be fun to teach people the basics of programming with it. It's really simple and introduces core concepts, but instead of things people don't understand like 'include' and '}' it uses 'CAN HAZ' and 'KTHXBYE'. The lightheartedness may have benefits.



  • Can I haz thread killd?
    Can not haz!
    SUDO DIE!!!!



  • Actually, I think error handling is done:

    AWSUM THX
        VISIBLE "YAYZ!"
    OH NOES
        VISIBLE "IOEGFOIEFEUFHHWE!"


  • @tarunik said:

    This is the same problem that is encountered by any code that depends on the current date.

    Sure, but it's really easy to solve outside the database. Just feed the method a mocked up datasource that barfs up a handful of carefully generated records.

    In the database, you have to purge the whole table and put in just the records you need. You also have to create any data necessary to satisfy database constraints. So, instead of just adding a few orders, you might be adding orders, customers, products, pricing structures, etc. If the database schema changes in a way unrelated to the functionality of this unit of code (another field is added to the table), then it could easily break the test.

    @tarunik said:

    With my approach it shouldn't be, as it'll give you a clean slate for each test case.

    But it's more work. Sometimes enough extra work that the effort makes the test not worth it.



  • @Jaime said:

    You don't see much for databases because DBMSs are more difficult to create automated unit test for because they tightly bind the method to the data.

    It depends on what exactly you're testing. You can unit test your schema, which doesn't have anything to do with concrete data (real or mocked).
    The post I replied to talks about SP's, so let's talk about that and consider my mentioning schema testing an aside.

    Why would the coupling between data and method/code be any tighter in a DBMS than in a regular program? SP's can take parameters and return values, so there is no need to pass data through globals (or a RDBM equivalent).
    SP implementations can be as self-contained as a C function, right?2

    @Jaime said:

    The problem with testing stored procedures is that you can't simply mock up the data storage and run the procedure, you have to actually store data, then run the procedure. This means you either have to run the procedure on a database that already has data in it - adding a dependency on that data, or create the data as a step in the test.

    ... just like you have to do when unit testing regular code.
    See @tarunik's remark about setUp() and tearDown() in comment <a href="http://what.thedailywtf.com/t/unit-testing-reserved-for-automation/4890/17"17.

    I'd even say that teardown after a unit test is easier with a database: wrap your data setup and test in a TRANSACTION/ROLLBACK and you don't have to worry about cleaning up state after your unit tests.

    @Jaime said:

    Procedures that are date dependent (like a hypothetical GetOrdersForCurrentMonth) become a challenge.

    If you don't write your code in a testable fashion, yes. That's not a problem unique to database tests though. If you write your code so (system) components can be easily mocked, it's not much of a challenge.

    @Jaime said:

    There is also the challenge of verifying that you are actually testing the right code. Pretty much the only way to do this is the have the test suite start with an empty database and pull the schema definitions from source control to create everything from scratch.

    I didn't know about it, but some googling reveals piggly:

    Piggly is a tool (written in Ruby, but you can write your tests in any language) to track code coverage of PostgreSQL PL/pgSQL stored procedures.

    That will tell you which branches of your DB code are exercised by your unit tests 😃

    @Jaime said:

    In other words - tools aren't the problem. The lack of tools simply represents the reality that testing at the database level is often more work than it's worth.

    I'd argue against the "than it's worth" part of your assertion. IM(HO|E) having unit tests with decent coverage are always1 worth the effort.

    I won't argue that it takes some work to set up a testing infrastructure though, but that's also not a problem unique to DB testing.
     
     
    1 usual disclaimer about using absolutes applies
    2 honest question: I can't immediately think of a counter-example to my statement, but I don't do much DB work



  • @OffByOne said:

    Why would the coupling between data and method/code be any tighter in a DBMS than in a regular program? SP's can take parameters and return values, so there is no need to pass data through globals (or a RDBM equivalent).SP implementations can be as self-contained as a C function, right?

    You're looking at the wrong side of the stored procedure. Sure, on the incoming side, stored procedures look very much like C functions. However, on the back end, there is no way to redirect a stored procedure to get mocked data instead of real data, so the stored procedure become bound to the data that it accesses. This is normally solved in code with Dependency Injection and/or Inversion of Control. There are no DI or IOC solutions for stored procedures.

    @OffByOne said:

    I'd even say that teardown after a unit test is easier with a database: wrap your data setup and test in a TRANSACTION/ROLLBACK and you don't have to worry about cleaning up state after your unit tests.

    Easier? in a well designed test using DI, you don't have to do anything to clean up. Also, Stateful methods should be a rarity, so state cleanup isn't a concern.

    @OffByOne said:

    That will tell you which branches of your DB code are exercised by your unit tests

    Piggly does nothing but measure code coverage. That doesn't address the issue I raised - you could still be testing different code than you thought you were testing.

    @OffByOne said:

    I'd argue against the "than it's worth" part of your assertion. IM(HO|E) having unit tests with decent coverage are always1 worth the effort.

    You would think. But the setUp() and tearDown() code ends up being fragile and most of the test failures will end up being false positives, leading to reduced effectiveness and additional maintenance of your test suite. Simply moving your code from stored procedure to a data access layer will drastically increase testability.

    @OffByOne said:

    If you write your code so (system) components can be easily mocked, it's not much of a challenge.

    So..... how exactly do you mock the data access behaviors of a stored procedure? You're answering my question by telling me to do the very thing that is the Achilles Heel of testing stored procedures. Once you move the logic out of the stored procedure, then you can mock stuff. If I translate your "write your code so (system) components can be easily mocked" to "rewrite your stored procedures as data layer methods", your statement becomes "If you rewrite your stored procedures as data layer methods, it's not much of a challenge.". You are arguing against your own point.


  • Java Dev

    All our db-related tests use a clean DB for each test, setup from scratch.

    A large segment of these tests use almost the same test code with different product configuration and input data to touch different parts of the code. We tend to call these 'pipeline tests', as they test data processing from end to end of the pipeline.



  • BIG FAT DISCLAIMER: I'm not an SQL dev. What I say is based on assumptions (that I try to verify by searching the web), the little DB dev work I did and having played with pgTAP not so long ago.
    I may very well be wrong, but at least I'm learning about a part of IT that I usually don't touch.
    Thanks @Jaime for your insightful replies!

    @Jaime said:

    You're looking at the wrong side of the stored procedure.
    [...]
    There are no DI or IOC solutions for stored procedures.

    Heh, you're right. I couldn't find anything on Google and I assume you know what you're talking about ;)

    I assumed that mocking system procedures (or calls to any external functions really) would be possible, even if the exact way to do that would be DBMS specific.

    That's actually a pity.

    @Jaime said:

    @OffByOne said:
    I'd even say that teardown after a unit test is easier with a database: wrap your data setup and test in a TRANSACTION/ROLLBACK and you don't have to worry about cleaning up state after your unit tests.

    Easier? in a well designed test using DI, you don't have to do anything to clean up. Also, Stateful methods should be a rarity, so state cleanup isn't a concern.

    Ok, think I wrote "persistent changes made to the dataset by the SP under test" instead of "state", since that was what I meant (but chose the wrong wording for).

    @Jaime said:

    @OffByOne said:
    That will tell you which branches of your DB code are exercised by your unit tests

    Piggly does nothing but measure code coverage. That doesn't address the issue I raised - you could still be testing different code than you thought you were testing.

    Howso? If generate a coverage report for your unit test, surely you can verify the code paths you expect to be tested are executed, nothing more, nothing less?

    @Jaime said:

    the setUp() and tearDown() code ends up being fragile and most of the test failures will end up being false positives, leading to reduced effectiveness and additional maintenance of your test suite. Simply moving your code from stored procedure to a data access layer will drastically increase testability.

    That's a problem with unit testing in general, not just for DB testing. GIGO and all that.
    If you don't maintain your test suite (and change tests as the code base under test evolves), getting less than stellar test results is to be expected.

    I can see how moving code from SP to DAL can improve testability, especially if your DAL is written in a language that's more flexible wrt mocking, IoC and DI.

    @Jaime said:

    So..... how exactly do you mock the data access behaviors of a stored procedure? You're answering my question by telling me to do the very thing that is the Achilles Heel of testing stored procedures.

    My Big Ass Assumption was wrong. I thought it was possible to override system procedures by mucking around in the system tables or something like that.

    I didn't realize such a crucial part of most modern applications is such a bitch to test. I seemed reasonable to believe that kind of thing would be better supported.



  • @OffByOne said:

    I didn't realize such a crucial part of most modern applications is such a bitch to test.

    Welcome to the life of anyone that regularly interacts with successful cowboy coders. I'm fine with people using stored procedures for data access logic if there is a defensible reason for it (like isolating applications from schema when the schema might be modified by several application teams). As soon as people start suggesting that stored procedures be used for business logic, I start asking these questions. The answer will inevitably be "I've been doing this for twenty years and it's always worked for me, back off". This roughly translates to "I don't care to participate in the maturation of my profession that has happened over the past twenty years".

    BTW, testing isn't the only problem with putting application logic in the database.



  • @Jaime said:

    @OffByOne said:
    I didn't realize such a crucial part of most modern applications is such a bitch to test.

    Welcome to the life of anyone that regularly interacts with successful cowboy coders.

    I don't get what cowboy coders have to do with DB's being difficult to test (because of a lack of IoC and/or DI).
    I do get why cowboy code is not fun to maintain though 😄

    @Jaime said:

    I'm fine with people using stored procedures for data access logic if there is a defensible reason for it (like isolating applications from schema when the schema might be modified by several application teams).

    Sure.
    Also, using an SP to perform some extra massaging/filtering of the resultset that can't be easily achieved in plain SQL seems like another good reason. Pumping a few million rows from the DB to the application, so it can filter out the 5 rows you're interested isn't such a stellar idea.

    @Jaime said:

    As soon as people start suggesting that stored procedures be used for business logic, I start asking these questions. The answer will inevitably be "I've been doing this for twenty years and it's always worked for me, back off". This roughly translates to "I don't care to participate in the maturation of my profession that has happened over the past twenty years".

    Heh. Appeal to authority is a logical fallacy. Appeal to your own authority just throws some Dunning-Kruger in the mix.

    @Jaime said:

    BTW, testing isn't the only problem with putting application logic in the database.

    No argument there. That's as bad as putting logic in your UI layer.



  • @OffByOne said:

    I don't get what cowboy coders have to do with DB's being difficult to test

    Cowboy coders only consider the question "does it work?" They tend to use untestable solutions more than mature coders do.

    @OffByOne said:

    Also, using an SP to perform some extra massaging/filtering of the resultset that can't be easily achieved in plain SQL seems like another good reason.

    There is literally nothing that can be done in a stored procedure that cannot be done without a stored procedure. Choosing to use a stored procedure is a choice of where the code is stored, not what it does.

    @OffByOne said:

    Pumping a few million rows from the DB to the application, so it can filter out the 5 rows you're interested isn't such a stellar idea

    That's a red herring. If a code-based data access layer pulls in a few million rows and throws away all but 5, it's poorly coded. So, you are comparing a properly written stored procedure to a poorly written hypothetical chunk of code - of course the properly written stored procedure is better. That's not an argument for stored procedures, it's an argument for not writing code poorly.



  • Business logic is evil in SQL.

    Testing aside.

    It is so complicated, apparently, that I have yet to work at a shop that doesn't manage to have it constantly falling apart.

    Hell, even creating a search engine that supports 20 possible search fields and performing the search with joins and views in SQL always turns into a performance issue for some reason.

    Simple joins created because you want to store reusable data in a separate table is ok, but once you start using joins for complex lookups, then have to collapse the joins to return a distinct set, you just broke your shit.


  • I survived the hour long Uno hand

    I didn't mean to imply that there are not tools, merely that we do not have them available at the moment. I'm pushing for us to update our Redgate developer bundles to the latest version, which includes unit testing tools.

    @Jaime said:

    The lack of tools simply represents the reality that testing at the database level is often more work than it's worth.

    Redgate's got a tool that automatically makes a temporary clone of any table you ask it to mock so you can insert just the data you need for the test. It's better than doing it by hand, for sure, and if you have business logic in the database it seems worth testing.

    @xaade said:

    Business logic is evil in SQL.

    True that. But it's easier to slide in tests for legacy code than to get it rewritten.



  • In the case of business logic in SQL, I'm going to have to disagree.

    It's a stretch, but I don't want to think of any case where it's better just to leave the SQL.



  • @Yamikuronue said:

    Redgate's got a tool that automatically makes a temporary clone of any table you ask it to mock so you can insert just the data you need for the test. It's better than doing it by hand, for sure, and if you have business logic in the database it seems worth testing.

    I was doing something similar back in 2006. I wrote a tool that allowed you to select rows and it generated the code to insert that data. I used the tool for the setup of my unit tests and tested my stored procedures. Doing it isn't the problem. The problem is that every major coding effort broke a handful of these tests and 90% of the time, the failure was solved by fixing the test rather than a bug in the code.


  • I survived the hour long Uno hand

    @Jaime said:

    The problem is that every major coding effort broke a handful of these tests and 90% of the time, the failure was solved by fixing the test rather than a bug in the code.

    You mean, when you intentionally changed things, the executable documentation indicating how it was supposed to work needed to be updated too? Le gasp! How horrific that must have been! Obviously test cases should Just Know what you mean the code to do.



  • @Yamikuronue said:

    You mean, when you intentionally changed things, the executable documentation indicating how it was supposed to work needed to be updated too? Le gasp! How horrific that must have been! Obviously test cases should Just Know what you mean the code to do.

    IMO, the executable documentation should be updated first...


  • ♿ (Parody)

    @Jaime said:

    The problem is that every major coding effort broke a handful of these tests and 90% of the time, the failure was solved by fixing the test rather than a bug in the code.

    I would expect this sort of thing (assuming the major effort didn't already include updating the tests). I mean...if your tests didn't need to change for something major then I'd assume your tests weren't very useful to begin with.



  • @Yamikuronue said:

    Obviously test cases should Just Know what you mean the code to do.

    What I'm saying is that stored procedure tests were much harder to minimize dependencies and side effects. Many more sp tests had false positive failures than code tests. Your saying "it happens" doesn't address the fact that it happened at a much greater rate with these tests.



  • @boomzilla said:

    I would expect this sort of thing

    A good example was that a setup routine would sometimes break due to schema changes unrelated to the functionality being tested. If this happened in code, the test suite would throw a bunch of compile errors and we would have them fixed in minutes, before the first test failed. We didn't know the sp tests failed until we ran them. After they failed, the structure of the setup code made it take longer to identify the problem and fix it. Even with the RedGate tool mentioned above, you would still have to go through some steps to regenerate the test data blob that would add to the time taken to get the test up and running.



  • @Jaime said:

    A good example was that a setup routine would sometimes break due to schema changes unrelated to the functionality being tested. If this happened in code, the test suite would throw a bunch of compile errors and we would have them fixed in minutes, before the first test failed. We didn't know the sp tests failed until we ran them. After they failed, the structure of the setup code made it take longer to identify the problem and fix it. Even with the RedGate tool mentioned above, you would still have to go through some steps to regenerate the test data blob that would add to the time taken to get the test up and running.

    @chubertdev said:

    IMO, the executable documentation should be updated first...

    And that's what TDD is folks.
    Update the test (aka executable documentation, aka executable requirement specification), run the test, and modify the source until the test passes.



  • @xaade said:

    @chubertdev said:
    IMO, the executable documentation should be updated first...

    And that's what TDD is folks.
    Update the test (aka executable documentation, aka executable requirement specification), run the test, and modify the source until the test passes.

    Gives new meaning to the phrase "setting expectations"



  • Key phrase: "would sometimes break due to schema changes unrelated to the functionality being tested"

    TDD doesn't mean visiting every test in the entire suite and trying to figure out which of the fragile ones are going to get pissed off today. The whole point is that TDD is more trouble than it's worth for deliverables that are hard to test. Another way to say it is that TDD would suggest that you refactor code so that it's more testable - e.g. move logic in stored procedures to a more testable technology.


  • Discourse touched me in a no-no place

    But the correct locus for the logic depends on the data, and always did (because it's much easier to move code around than data once the data size starts to go up). Forcing a remote database to send you a million rows just so you can apply a complex filter to get the 5 you care about in order to keep that filtering logic testable will get you a starring role on the main TDWTF site. If that makes your logic hard to test, well boo hoo: learn to test better! Setting up a database per test is entirely reasonable if that's what it takes to ensure that the code in question is going to function correctly and robustly.

    Remember, one man's integration test is another man's unit test, because one man's complex software product is another man's plug-in module. And if you've done everything via injected mocks, you've always got the worry that the mocks aren't going to behave like the thing that they're simulating in some important way…

    For all the above, “test first, then code to satisfy the test” remains a good principle. It gives you a way to measure when you're done.



  • So you're saying that sometimes there are exceptions. So what? The point of focusing on testability is so that you make everything that can be easily testable easy to test. Obviously, there will be some cases where doing something different is important as a performance optimization.

    There are two possible takeaways from your post:

    1. You should generally do your filtering in the database because it's faster, even if it makes testing harder.
    2. Sometimes you should not blindly follow a set of rules because a situation requires it.

    If you meant #1, then you are one of the cowboy coders mentioned above and a classic example of premature optimization.

    If you meant #2, then you are stating the obvious, but doing it in a way that will cause cowboy coders to read it as if you meant #1 and will use your post as another mental building block in their wall of ignorance.



  • @Jaime said:

    If you meant #1, then you are one of the cowboy coders mentioned above and a classic example of premature optimization.

    No, this is using the correct algorithm for the job levels of optimization, not the micro-optimizations referred to by those who detest "premature optimization". Trying to do query filtering in code qualifies as "reinventing a square wheel", even, especially when joins and complex conditions are involved.

    @dkf said:

    Setting up a database per test is entirely reasonable if that's what it takes to ensure that the code in question is going to function correctly and robustly.

    Exactly -- and for those types (probably DBAs) who go 'boo hoo -- how are we supposed to keep track of a zillion dev database schemas now?', my answer is twofold:

    1. Let your developers run local -- or at the very least independent -- environments of some sort instead of relying on a centrally located/shared 'dev environment' -- that notion is a silly piece of bean-counter penny-pinching!
    2. Automate it! Ideally, 'push butan, get pre-cloned-from-a-clean-image dev database' would be a thing in your world.

    @dkf said:

    And if you've done everything via injected mocks, you've always got the worry that the mocks aren't going to behave like the thing that they're simulating in some important way…

    THIS! You need the ability to integration test quickly in a mockist environment to catch issues where your mocks aren't mocking properly -- especially if they are mocking complex third-party subsystems.

    Filed under: Would you filter a seven-table join in code?


  • Discourse touched me in a no-no place

    I'm pointing out that testability is not the only valuable metric, and in fact it is one of the metrics that it is least easy to quantify the end-user-visible utility of. Yes, it's a good idea anyway, but that's because it leads to faster development of quality code, and that in turn leads to potentially better satisfaction of things that the users really do care about.

    Filtering a result set should usually be done in the database precisely because it reduces the amount of traffic over an interface that has a very long history of being a key bandwidth bottleneck. It doesn't need to be the full filtering, but you do want to get from millions down to hundreds; that four orders of (decimal) magnitude1 will do wonders for the overall performance of the application, and that is something that is an end-user-observable. Users might care about correctness more than speed, but they still care about speed a lot.

    Speed is an awkward thing to test, especially with unit tests, since it is intimately connected with the deployment configuration. This proves that TDD is an incomplete methodology for software development. (BDD is slightly better because it has a wider concept of observable that can correspond more strongly with reasonable project requirements.)

    1 Or about 9.2 orders of magnitude in God's own number base, e, which everyone knows about, right?


  • Discourse touched me in a no-no place

    @tarunik said:

    Automate it! Ideally, 'push butan, get pre-cloned-from-a-clean-image dev database' would be a thing in your world.

    If you can, working with SQLite makes dealing with this sort of thing extremely easy. Clone some database to make a local throwaway instance? Copy a file to a temporary location. Job done. (I also have a pretty high level of confidence that SQLite will be functioning as described; I know quite a bit about the testing regime that its developers use for it…)



  • @tarunik said:

    Trying to do query filtering in code qualifies as "reinventing a square wheel", even, especially when joins and complex conditions are involved.

    You do realize that the best solutions accomplish both at the same time - filtering by specifying the filter in code, but yet still execute the filter in the database engine. Us "people who do it in code" don't ever fetch data then filter it - that's just a straw man for the stored procedure advocates to beat on.

    Also, this should all be in the data access layer, where there is no business logic, so there isn't much to test.

    Also, if your data access layer is doing complex joins, then it is probably doing business logic and you already have an object model problem. You are well on your way to making a distinct DAL object for every UI.



  • @dkf said:

    If you can, working with SQLite makes dealing with this sort of thing extremely easy. Clone some database to make a local throwaway instance?

    Here we are back to making the "how to do it" easy, and re-creating the fragility problem that comes with having such a large and complex dependency (a whole database in your example).



  • @dkf said:

    If you can, working with SQLite makes dealing with this sort of thing extremely easy. Clone some database to make a local throwaway instance? Copy a file to a temporary location. Job done. (I also have a pretty high level of confidence that SQLite will be functioning as described; I know quite a bit about the testing regime that its developers use for it…)

    THIS! Is there some sort of wall of disbelief that people have that says "no, no, there's no giant server with a DBA behind it, so it can't be a database" about SQLite? It's one of the best-tested database engines in the world I reckon, and also the most widely deployed one out there by a large margin -- the SQLite team conservatively estimates that half a billion copies of it are in circulation, and I have no reason not to believe them!

    About the only thing that keeps it from being a universal solution to the test-database problem is the unwillingness of

    1. The SQL standard committee to approach some problems (such as windowed queries -- yes, these make sense in some applications) sanely, while proceeding to inject every last piece of bloat they could get their hands on into the SQL standard, and
    2. database vendors to tighten up their support for the "core" of the SQL standard -- MySQL, I'm looking squarely at you here...

Log in to reply