You mean the data needs to be consistent?



  • A month ago one of our more intelligent animals morons retards users gave me a spreadsheet with a bunch of columns of sample data to be used to feed a script to do a bulk update of some data in the db. Ok, I write the script. I test the script. HE tests the script.

    A week ago he gives me an updated spreadsheet which I dutifully run through the script and verify that everything still works.

    We file the production update requests, get the requisite approvals, kiss all the pertinent asses to get permission to change the data.

    Ok, it's Friday night after work and I get the final - unannounced - unplanned - update to the spreadsheet. I smell a rat so I export the tabs to csv files and run it through the script in development before going near production. Every single sql statement it generates fails. A quick glance shows strings where numbers should be, numbers where string should be, dates where vectors should be... A few extra columns in the spreadsheet. Ok, remove the extra columns. Now we get the same thing, only in the other direction. Apparently some columns we were expecting but ignoring were removed. "But you didn't need that data; if it's missing, why would it cause a problem?"

    And of course, the link to the internal sharepoint where they stored the original spreadsheet no longer works.

    I go through the script and figure out what columns are supposed to have what data, and tell this .... user to give me the data in the correct format.

    While I'm grinding through all of this, he starts complaining that he has to go home (he starts at 9). I respond that I've been on since 6AM and I'D like to go home too.....

    Finally, he grudgingly gives me the data in the correct format and it goes right in.

    Then he calls a conference call with the (newly hired) big manager so he can complain that I took too long. I explain everything to the big guy, and then pose the question: how about from now on I have the application display the data in different columns of the JTables depending upon the day of the week? It's the same data, does it really matter where it is in the display table? The big boss got the point and suggested I go have a couple of drinks - on him (decent guy actually).

    Where do they get these people?

     



  • We've had these "import data from a spreadsheet into the DB" tasks before an the only way to do it is to take a copy of the final tested spreadsheet (e.g. on a memory stick) and use that. If you don't lock it down then the users are guranteed to change and break it.

    Nice work from the boss though - the opposite of the stipulated lunch guy.

    @snoofle said:

    Where do they get these people?

    The user community has an endless supply of these.



  • @snoofle said:

    The big boss got the point and suggested I go have a couple of drinks - on him
     

    :O



  • @snoofle said:

    he starts complaining that he has to go home.

    <snip>

    Then he calls a conference call with the (newly hired) big manager so he can complain that I took too long.

    Never too late to be a prick, is it?

    Incidentally do you have any Pentium II machines?



  • Speaking of fun "import excel into database" projects, I've recently stumbled over Google Refine, which is an program that's supposed to help clean and correct messy and inconsistent data in (among other formats) text files and excel tables.
    It basically tries to treat the file as a single database table and gives you various options to filter columns or transform their values. All changes are shown directly but can be undone, so you are engouraged to "play around". I admint, I haven't tried it out on a real project yet, but it looks like it could be pretty useful.

    Minor WTF: Because they want you to to be able to use the program on sensitive data, this is an actual desktop application and not "in the cloud". But google still holds on to their "HTML/CSS/JS are teh future of programming" paradigm. How does this work? Simple: Google Refine is a GWT web app - hosted on it's own web server (stripped-down jetty) that is installed together with the software...



  • @PSWorx said:

    Speaking of fun "import excel into database" projects, I've recently stumbled over Google Refine, which is an program that's supposed to help clean and correct messy and inconsistent data in (among other formats) text files and excel tables.
    It basically tries to treat the file as a single database table and gives you various options to filter columns or transform their values. All changes are shown directly but can be undone, so you are engouraged to "play around". I admint, I haven't tried it out on a real project yet, but it looks like it could be pretty useful.

    I usually save to tab-delimited and use awk to "massage" the data before putting it into the database. It's great for things like seeing if every row has the same number of columns, forcing dates to be formatted as dates, etc.

    @PSWorx said:

    Minor WTF: Because they want you to to be able to use the program on sensitive data, this is an actual desktop application and not "in the cloud". But google still holds on to their "HTML/CSS/JS are teh future of programming" paradigm. How does this work? Simple: Google Refine is a GWT web app - hosted on it's own web server (stripped-down jetty) that is installed together with the software...

    We had a big exec at our company send out a link to a "skills database" he's trying to build-- the link went to a Google Spreadsheet. Our lead IT guy replied, almost frothing at the mouth angry, that he was encouraging employees to give up all our company data to Google. The sheet was taken down. The dumbest part is that we have a Sharepoint server, so he could have just made the same fucking site on Sharepoint and nobody would have complained.



  • @blakeyrat said:

    ...The dumbest part is that we have a Sharepoint server, so he could have just made the same fucking site on Sharepoint and nobody would have complained.
     

     

    Except about Sharepoint.



  • @dhromed said:

    @snoofle said:

    The big boss got the point and suggested I go have a couple of drinks - on him
     

    :O

     

    Then he called the jerk back and told him how tech people can be "difficult" and urged him to tread lightly around you in the future because even though he was right, as all business people always are by definition, it was a bad idea to endanger the company's relationship with someone who had such valuable knowledge and expertise and might retaliate in ways only other "difficult" tech people would be able to defend against.



  • @snoofle said:

    Where do they get these people?

     That is a good question. It is extremely hard to find a "decent guy"...especially one who is a "big manager" 


Log in to reply