Pointlessness



  • Today myself and another senior dev got a call from one of our client's sysadmins. Said sysadmin was atrempting to deploy the latest version of our app to a remote site, but things weren't working and he wondered if we could help? Well we tried, believe me, but as we asked questions to find out what was wrong we started to realise that maybe we weren't the ones who needed help.

    Firstly, the app in question is under constant development by a dedicated team of 12 devs (us) and an updated version goes live every 2 weeks (frontend and DB changes), but the latest version installed on the remote server was more than 2 months old. We (the devs) had been under the impression that all updates applied to the main server were also deployed to the remote server, but since the sysadmins are in charge of all deployments and had not said anything, we made the assumption that all deploys were happening and going fine. More fool us...

    Next we found that the sysadmin had overwritten the 2-month-old binaries with the latest ones but hadn't run any of the update scripts on the remote DB, hence massive breakage. (It didn't occur to him that the DB might need to be updated too.) Since the app modifies its database, it was now impossible to roll back to the previous binaries without updating the DB, so we all agreed that we might as well just fix the issues and get the latest there.

    We suggested that the sysadmin run the update scripts supplied with the last 2 months of releases, which he agreed sounded like a good idea, but unfortunately he wasn't sure exactly which scripts to run, nor in what order.

    We suggested he run all the scripts, starting with the oldest ones first, but after he got a single error (due to schema changes in the intervening period) he escalated the "issue" to his higher-ups because "we" had broken things. He made his first good decision of the day by reporting to a manager who knows that we are generally competent and don't fuck things up, and after that manager called us for the real story we were off the hook. However, the client still needed the broken remote deployment to be fixed.

    We suggested the sysadmin should run a DB "diff" utility on the local (up-to-date) database and the remote (broken) one and apply all the changes from local to remote, but that was rejected as too time-consuming and error-prone. We were grudgingly about to go back and dig through 2 months of source to find (and create) scripts for the remote DB, when the aforementioned manager spoke up:

    "But the remote users aren't even using that system right now; they're waiting for it to be brought up to date so they can be trained to use it. Why can't you just restore the local database over the remote one?"

    Which we (or rather, the sysadmin) could, and did. Hence a day's worth of wasted effort from 2 senior devs, plus much wailing from the sysadmin, was resolved by a 30-minute RESTORE DATABASE command, and there was once again peace in the land. However, I fear there's a permanent mark on my forehead due to the number of times my palm struck it today.

    We suggested that all deployments be kept in sync in the future. Let's see if that happens.



  • @The_Assimilator said:

    We suggested that all deployments be kept in sync in the future. Let's see if that happens.
     

    Can I suggest a table in your database called version, and that your application check this before starting up?



  • @Rick said:

    @The_Assimilator said:

    We suggested that all deployments be kept in sync in the future. Let's see if that happens.
     

    Can I suggest a table in your database called version, and that your application check this before starting up?

    Has already been proposed by multiple people multiple times. Unfortunately, client is not interested in paying for such functionality, and our regular deploys to dev/qa mean version number is yet another thing that has to be updated. And no, we can't check if the app was compiled in debug or release mode and disable/enable the version check accordingly, because the app only works when compiled in debug mode (we didn't write it, but we have to maintain it).



  • @The_Assimilator said:

    And no, we can't check if the app was compiled in debug or release mode and disable/enable the version check accordingly, because the app only works when compiled in debug mode (we didn't write it, but we have to maintain it).
    I'd inquire further, but at this point it's best not to ask.

     



  • @The_Assimilator said:

    because the app only works when compiled in debug mode

    Tell us more about this.



  • While the sysadmin doesn't sound like a very competent one, I must say that I notice a number of WTF's in your software & development schema...
    Is this software custom for said client or at least for a very small number of clients where you work closely with? Depending on the users (c.q. sysadmins) to always keep their software up-to-date is a giant WTF in itself...

    But maybe you can't help it, if you can't even get permission to add a version field to the database somewhere.



  • Sounds like you have a good manager, so that's nice.



  • @Sutherlands said:

    Sounds like you have a good manager, that's extremely rare, worth their weight in gold, and vital for you to keep a good relationship with them.

     

    FTFY.

     



  • Been there, felt your pain, solved it to a certain level by bagging up the changes into a bundle which can only be deployed by running a specific script which updates the database and updates the binaries all in one go, so the user isn't given the opportunity of making the mistake of doing the one but not the other. He has demonstrated that he is not computer literate, and so needs luser-level support. Giving written instructions doesn't work. You need to give him an app with a big red button that works once only.

    Seriously. There is no other way forward.


  • ♿ (Parody)

    @The_Assimilator said:

    Since the app modifies its database, it was now impossible to roll back to the previous binaries without updating the DB, so we all agreed that we might as well just fix the issues and get the latest there.

    Wait. Are you saying the app changes the schema? Or just data?



  • @The_Assimilator said:

    @Rick said:
    @The_Assimilator said:

    We suggested that all deployments be kept in sync in the future. Let's see if that happens.

    Can I suggest a table in your database called version, and that your application check this before starting up?

    Has already been proposed by multiple people multiple times. Unfortunately, client is not interested in paying for such functionality,

    Some features are for *you*, not the client, and the client shouldn't be the one paying.  It's a really simple feature.  It would save a lot of time in a case like this.  Unless you're able to bill hourly rates for your support in this sort of situation, I'd say it's a no-brainer.  Unfortunately, if you are billing hourly rates for that support, then your management will think it's a no-brainer to deny it, even if the client was willing to pay.  (Of course, if the client was demanding it, that could be another matter.)



  • @dtech said:

    While the sysadmin doesn't sound like a very competent one, I must say that I notice a number of WTF's in your software & development schema...
    Is this software custom for said client or at least for a very small number of clients where you work closely with? Depending on the users (c.q. sysadmins) to always keep their software up-to-date is a giant WTF in itself...

    But maybe you can't help it, if you can't even get permission to add a version field to the database somewhere.

    Custom software package for a single client, client's policy is that only their own on-site staff may do deployments, therefore responsibility to keep various installations up-to-date is the client's.

    @boomzilla said:

    Wait. Are you saying the app changes the schema? Or just data?

    Both.



  • @tgape said:

    @The_Assimilator said:
    @Rick said:
    @The_Assimilator said:

    We suggested that all deployments be kept in sync in the future. Let's see if that happens.

    Can I suggest a table in your database called version, and that your application check this before starting up?

    Has already been proposed by multiple people multiple times. Unfortunately, client is not interested in paying for such functionality,

    Some features are for *you*, not the client, and the client shouldn't be the one paying.  It's a really simple feature.  It would save a lot of time in a case like this.  Unless you're able to bill hourly rates for your support in this sort of situation, I'd say it's a no-brainer.  Unfortunately, if you are billing hourly rates for that support, then your management will think it's a no-brainer to deny it, even if the client was willing to pay.  (Of course, if the client was demanding it, that could be another matter.)

     

    Disagree strongly - all but the most brain-dead gibbering monkey for management will recognise that a small 'in house' feature to prevent this kind of scenario, is worth more in sales/quality value then the possible value of billed hours to sort out a shit heap situation.  If your management really believe that doing a SHIT job is the way to garantee future work then you should spit in their face and tell them it will save on water utility bills for a shower.

     



  •  @The_Assimilator said:

    @Rick said:

    @The_Assimilator said:

    We suggested that all deployments be kept in sync in the future. Let's see if that happens.
     

    Can I suggest a table in your database called version, and that your application check this before starting up?

    Has already been proposed by multiple people multiple times. Unfortunately, client is not interested in paying for such functionality, and our regular deploys to dev/qa mean version number is yet another thing that has to be updated. And no, we can't check if the app was compiled in debug or release mode and disable/enable the version check accordingly, because the app only works when compiled in debug mode (we didn't write it, but we have to maintain it).

    You are not selling a version number in the database, you are selling a error-free update system. Seems trivial to me to claim that the next update cost 1-2 hours additional development to prevent update errors that took 3 people a full day work last update.

     



  • @havokk said:

    @Sutherlands said:

    Sounds like you have a good manager, that's extremely rare, worth their weight in gold, and vital for you to keep a good relationship with them.

     

    FTFY.

     

    This.  I'd throw myself in front of a car for my CTO.

     



  • @Matt Westwood said:

    Been there, felt your pain, solved it to a certain level by bagging up the changes into a bundle which can only be deployed by running a specific script which updates the database and updates the binaries all in one go, so the user isn't given the opportunity of making the mistake of doing the one but not the other. He has demonstrated that he is not computer literate, and so needs luser-level support. Giving written instructions doesn't work. You need to give him an app with a big red button that works once only.

    Seriously. There is no other way forward.
    +100, this is the solution.



  • @hoodaticus said:

    @Matt Westwood said:

    Giving written instructions doesn't work. You need to give him an app with a big red button that works once only.

    Seriously. There is no other way forward.
    +100, this is the solution.

     

    Hell, even if the person WAS computer-literate, providing a big red button pleases even the most hardcore unix-sysadmins. It means they can get on with their job instead of doing your busywork.

     



  • @havokk said:

    @Sutherlands said:

    Sounds like you have a good manager, that's extremely rare, worth their weight in gold, and vital for you to keep a good relationship with them.

     

    FTFY.

     

    QFT



  • @Helix said:

    @tgape said:

    Some features are for you, not the client, and the client shouldn't be the one paying.  It's a really simple feature.  It would save a lot of time in a case like this.  Unless you're able to bill hourly rates for your support in this sort of situation, I'd say it's a no-brainer.  Unfortunately, if you are billing hourly rates for that support, then your management will think it's a no-brainer to deny it, even if the client was willing to pay.  (Of course, if the client was demanding it, that could be another matter.)

     

    Disagree strongly - all but the most brain-dead gibbering monkey for management will recognise that a small 'in house' feature to prevent this kind of scenario, is worth more in sales/quality value then the possible value of billed hours to sort out a shit heap situation.  If your management really believe that doing a SHIT job is the way to garantee future work then you should spit in their face and tell them it will save on water utility bills for a shower.

     

    Both of you are missing the point. This is something which you don't go to the client to get approval for. It's not even something you ask your manager for permission to do. It's part of your task to design an upgrade program / script / etc. because it's part of your job.



  • @Matt Westwood said:

    @Helix said:

    @tgape said:

    Some features are for *you*, not the client, and the client shouldn't be the one paying.  It's a really simple feature.  It would save a lot of time in a case like this.  Unless you're able to bill hourly rates for your support in this sort of situation, I'd say it's a no-brainer.  Unfortunately, if you are billing hourly rates for that support, then your management will think it's a no-brainer to deny it, even if the client was willing to pay.  (Of course, if the client was demanding it, that could be another matter.)

     

    Disagree strongly - all but the most brain-dead gibbering monkey for management will recognise that a small 'in house' feature to prevent this kind of scenario, is worth more in sales/quality value then the possible value of billed hours to sort out a shit heap situation.  If your management really believe that doing a SHIT job is the way to garantee future work then you should spit in their face and tell them it will save on water utility bills for a shower.

     

     

    Both of you are missing the point. This is something which you don't go to the client to get approval for. It's not even something you ask your manager for permission to do. It's part of your task to design an upgrade program / script / etc. because it's part of your job.

     

    Sorry to nitpick, but I would do it because it would make my life easier. A minor side effect is that it would be better for both vendor and customer and part of my job.



  • @Rick said:

    @Matt Westwood said:

    @Helix said:

    @tgape said:

    Some features are for *you*, not the client, and the client shouldn't be the one paying.  It's a really simple feature.  It would save a lot of time in a case like this.  Unless you're able to bill hourly rates for your support in this sort of situation, I'd say it's a no-brainer.  Unfortunately, if you are billing hourly rates for that support, then your management will think it's a no-brainer to deny it, even if the client was willing to pay.  (Of course, if the client was demanding it, that could be another matter.)

     

    Disagree strongly - all but the most brain-dead gibbering monkey for management will recognise that a small 'in house' feature to prevent this kind of scenario, is worth more in sales/quality value then the possible value of billed hours to sort out a shit heap situation.  If your management really believe that doing a SHIT job is the way to garantee future work then you should spit in their face and tell them it will save on water utility bills for a shower.

     

     

    Both of you are missing the point. This is something which you don't go to the client to get approval for. It's not even something you ask your manager for permission to do. It's part of your task to design an upgrade program / script / etc. because it's part of your job.

     

    Sorry to nitpick, but I would do it because it would make my life easier. A minor side effect is that it would be better for both vendor and customer and part of my job.

     

    That was my point - Both of these reasons of motivation to do the work unbilled will have a outcome of improving overall quality of service. Assuming that if you feel better about your life, you tend to work a little more positively..

     


Log in to reply