Those unstable databases



  • I used to work at a radio station, and part of my tasks included keeping the rather old automation system up and running. Now granted, this system was a 98 machine running on a beige box 500mhz and 128mb of ram, but the software was also a little at fault for the crashing it constantly was experiencing.

     The college I worked for also employed a guy who had been gang pressed into fixing problems at the station from time to time. So when I came on staff he was given the task of getting me up to speed. We went over the automation system and we both complained about the fact that the logs and schedules were all stored in Access databases, but what could we do? 

     A few months later I found myself talking to "Jerry" (as we'll call this guy). And discussing with him how unstable the current system was. I meantioned that I was thinking of switching to Rivendell (a GPL automation system). The conversation went something like this:

    Me: "The current system is trash, it crashes all the time, I'm thinking of switching to some newer software."

    Jerry: "You know, the guy who ran the station before you came is writing his own software."

    Me: "Really?"

    Jerry: "Yeah, he's using VB."

    Me: "Ah huh. Well, I've been looking at Rivendell, it looks good. And it even keeps all the data in a MySQL database."

    Jerry: "Oh that's no good. This other program uses flat files which are way more stable."

    Me: "More stable?"

    Jerry: "Yeah, databases cause lots of headaches. They crash all the time, corrupt data, etc. Using text files is better."

    Me: "Banks use databases to store data."

    Jerry: "Yeah, and you know how often those systems crash?"

    <silence> 

     

    How do you answer that? Needless to say, I kindof stopped asking for his advice on things.



  • Too bad you can't express "lol wut?" properly in a verbal conversation. 



  • I work for a bank.  Trust me when I say he's right.



  •  For a high volume/usage database maybe. But I've worked on dozens of SQL driven websites, and never had a single one crash. 



  • <agree>

    @vt_mruhlin said:

    I work for a bank.  Trust me when I say he's right.

    </agree>

    However  with databases there are all sort of solutions like: High Availability, Load Balancing, Good programming practices (except oracle, oracle just crashes :P)

    The other point of data corruption, I would assume that databases actually corrupt data less often than flat files with a custom system written by a dude and VB... Unless heavilty QAed.

    For performance, my co-worker uses nifty binary search algorithms to write a flat-file based search which searches at speeds no database can match, because he knows the format of all data and looks for specifics. Very easy to manage as well. But does not offer the flexibility of a DB, but at that point he don't care.

    Just to add the OP co-worker's point's weight: Databases can be difficult to manage. To add the OP's argument, I don't think VB is a good tool for making efficient databases.



  • @dlikhten said:

    a flat-file based search which searches at speeds no database can match, because he knows the format of all data and looks for specifics. Very easy to manage as well.

    I think we found the 'swampie' that SpectateSwamp is always referring to.

    Keep up the good work at the SwampShack!



  • @dlikhten said:

    The other point of data corruption, I would assume that databases actually corrupt data less often than flat files with a custom system written by a dude and VB... Unless heavilty QAed.

    Back in the 90's that was called DBase, and later, BTrieve.  The #1 problem was corrupted indexes due to multiple users modifying the same data files at the same time.  There were two workarounds 1) reindex all of the database files nightly and fix any corruption along the way; or, 2) buy a real database server. 

    Granted, it wasn't written in VB, but it was the exact same idea.

     



  • @clively said:

     BTrieve.

     

     

    Ahem....we like to refer to that as "The fucked up system that must not be named."

    Pervasive is FTL. 



  • The UnKnowAbles

    @MasterPlanSoftware said:

    @dlikhten said:

    a flat-file based search which searches at speeds no database can match, because he knows the format of all data and looks for specifics. Very easy to manage as well.

    I think we found the 'swampie' that SpectateSwamp is always referring to.

    Keep up the good work at the SwampShack!

    Whooo Hooo. I'm Not Alone.  Simple text flat files. That's the way to go. Easy to view and fix/manipulate/rig you name it. And we need to know way way less than the rest.


  • Considered Harmful

    @SpectateSwamp said:

    @MasterPlanSoftware said:

    @dlikhten said:

    a flat-file based search which searches at speeds no database can match, because he knows the format of all data and looks for specifics. Very easy to manage as well.

    I think we found the 'swampie' that SpectateSwamp is always referring to.

    Keep up the good work at the SwampShack!

    Whooo Hooo. I'm Not Alone.  Simple text flat files. That's the way to go. Easy to view and fix/manipulate/rig you name it. And we need to know way way less than the rest.

    Has no one realized this guy is just a troll? I can't believe his Desktop Search thread has gotten so many replies. There's no way he could honestly believe that his program is comparable to any real solution on the market. I knew as soon as I saw the source code it was a joke.



  • @joe.edwards@imaginuity.com said:

    Has no one realized this guy is just a troll? I can't believe his Desktop Search thread has gotten so many replies. There's no way he could honestly believe that his program is comparable to any real solution on the market. I knew as soon as I saw the source code it was a joke.
     

    Thanks for the insight, you have opened my eyes to the fact that he is useless!



  •  Huh, what? I started the desktop search thread? oook.

    No, what i was saying is: for very specific solutions sometimes a database is not the right way to go about things. A WTF is to think that database solves all isues period. I am saying there are some cases where a flat file with indexing based on the fact that you know you can do a binary search on say the first 10 characters of every line may be very efficient, and not require the maintenance overhead of a database. However very rarely do I find these to be the case.

    The argument that:

    1) Databases are super-stable

    and

    2) Databases solve all our data issues perfectly

    is bad.

    For DBs to be stable you need competent DBAs.Sometimes you have them, sometimes you don't. True the argument here that databases are always unstable is a bad one. And I assure you we have seen major performance increases from using a regex on a flat file than doing a database query because query might do a table-scan on 130million records, instead of just looking at the last 100 lines of a log file which is a bit more efficient no?

    In any case i am not swampie, or SpectateSwamp or w/e. And I am not trolling anyone (intentionally)



  • @dlikhten said:

     Huh, what? I started the desktop search thread? oook.

    No, what i was saying is: for very specific solutions sometimes a database is not the right way to go about things. A WTF is to think that database solves all isues period. I am saying there are some cases where a flat file with indexing based on the fact that you know you can do a binary search on say the first 10 characters of every line may be very efficient, and not require the maintenance overhead of a database. However very rarely do I find these to be the case.

    The argument that:

    1) Databases are super-stable

    and

    2) Databases solve all our data issues perfectly

    is bad.

    For DBs to be stable you need competent DBAs.Sometimes you have them, sometimes you don't. True the argument here that databases are always unstable is a bad one. And I assure you we have seen major performance increases from using a regex on a flat file than doing a database query because query might do a table-scan on 130million records, instead of just looking at the last 100 lines of a log file which is a bit more efficient no?

    In any case i am not swampie, or SpectateSwamp or w/e. And I am not trolling anyone (intentionally)

     

    Sounds to me like SpectateSwamp has brought you over to his side. Soon you will be displaying your random data on the screen with the Desktop Search pro!

    First you must finish merging all your files together into a single text dump and screen capping all your other non-text documents. Then you get a well deserved break crawling through the brush with Spectate filming squirrels.



  • I maintained a couple dozen MySQL databases for about 3 years.  They corrupted themselves more than a few times.  In some cases, faulty disks and power failures were probably to blame.

     

    However, a flat-file based system would have corrupted itself just as often, and probably more often, when dealing with the same transaction volume.



  • Re: Spectate hijacked the thread's title.

    @merreborn said:

    I maintained a couple dozen MySQL databases for about 3 years.  They corrupted themselves more than a few times.  In some cases, faulty disks and power failures were probably to blame.

     

    However, a flat-file based system would have corrupted itself just as often, and probably more often, when dealing with the same transaction volume.

     

    Not to mention scaling would be some kind of distant dream...



  • @Lysis said:

    Too bad you can't express "lol wut?" properly in a verbal conversation. 

     

    No, but you can get a fairly close approximation with the proper arch of an eyebrow. 



  • @dlikhten said:

    I am saying there are some cases where a flat file with indexing based on the fact that you know you can do a binary search on say the first 10 characters of every line may be very efficient,...

    Any reasonably competent DB can do exactly that. And correctly -- the first time. Without having to re-invent it again.

    @dlikhten said:

    and not require the maintenance overhead of a database.

    What maintenance overhead? How can a [good] database with built-in solutions for maintenance be more trouble than rolling it all yourself?


  • Discourse touched me in a no-no place

    @dlikhten said:

    For performance, my co-worker uses nifty binary search algorithms to write a flat-file based search which searches at speeds no database can match, because he knows the format of all data and looks for specifics. Very easy to manage as well. But does not offer the flexibility of a DB, but at that point he don't care.

    I didn't see you volunteered, or vounteering.

    How is the swamp?

    Or are we being meta-trolled?</rhetorical>



  • Not sure what the relationship between VB and flat files is. I've got VB happilly chatting with Oracle over here.

     



  •  If databases "crash all the time" then all you'd need to say is Windows crashes all the time. Rivendell looks interesting as it runs on Linux.



  • @MasterPlanSoftware said:

    Not to mention scaling would be some kind of distant dream...

     

    Yeah, because we all know every system ever written needs to be scalable "just in case".




  • @Quinnum said:

    @MasterPlanSoftware said:

    Not to mention scaling would be some kind of distant dream...

     

    Yeah, because we all know every system ever written needs to be scalable "just in case".

     

    ... I detect sarcasm... thinking that the apps you write shouldn't be made scalable by default is dangerous thinking and is likely the cause of many of the WTFs on this site.

    Just ask ObiWayne...



  • @SpectateSwamp said:

    Simple text flat files. That's the way to go. Easy to view and fix/manipulate/rig you name it. And we need to know way way less than the rest.
     

     

    Oof.  Must resist urge to launch into yet another SSDS flame.  Okay, taking this in a more general context:

     

    For any project involving a significant amount of regularized data, the benefit of a RDBMS is well worth the overhead of communication and defining tables.

     

    Flat files require you to pay attention to low-level details like "what column does such-and-such field start in?".  Sure, you can do it, but anyone with a clue would prefer not to have to.

     

    When it comes to financial and inventory systems (my bread and butter), arbitrary end-user manipulation of the data is absolutely the last thing you want.  Double-entry bookkeeping would be the first thing to go, and then it would all end in tears, I tell you what.

     



  • @MasterPlanSoftware said:

    @Quinnum said:

    @MasterPlanSoftware said:

    Not to mention scaling would be some kind of distant dream...

     

    Yeah, because we all know every system ever written needs to be scalable "just in case".

     

    ... I detect sarcasm... thinking that the apps you write shouldn't be made scalable by default is dangerous thinking and is likely the cause of many of the WTFs on this site.

    Just ask ObiWayne...

     

    So you are the thought police now ?

    No, the cause of many of the WTFs on this site is just poor logic or programming ability in the first place.

    Did you ever hear about YAGNI? Thinking that every app you write automatically need to be so precisely over-engineered for features that may or may not ever be actually required is the reason most systems are the bloated pieces of shit they currently are, and also the cause of many other WTFs on this site.

    Yeah, that single user app for keeping my DVD collection needs to be scalable. That little PDA application that requires some simple small set of data needs to be scalable.

    The right tool for the right situation. Surely you have been here long enough to know that shoe-horning everything into a single paradigm or methodology is not appropriate. Everything is not a nail.



  • @Quinnum said:

    @MasterPlanSoftware said:

    @Quinnum said:

    @MasterPlanSoftware said:

    Not to mention scaling would be some kind of distant dream...

     

    Yeah, because we all know every system ever written needs to be scalable "just in case".

     

    ... I detect sarcasm... thinking that the apps you write shouldn't be made scalable by default is dangerous thinking and is likely the cause of many of the WTFs on this site.

    Just ask ObiWayne...

     

    So you are the thought police now ?

    No, the cause of many of the WTFs on this site is just poor logic or programming ability in the first place.

    Did you ever hear about YAGNI? Thinking that every app you write automatically need to be so precisely over-engineered for features that may or may not ever be actually required is the reason most systems are the bloated pieces of shit they currently are, and also the cause of many other WTFs on this site.

    Yeah, that single user app for keeping my DVD collection needs to be scalable. That little PDA application that requires some simple small set of data needs to be scalable.

    The right tool for the right situation. Surely you have been here long enough to know that shoe-horning everything into a single paradigm or methodology is not appropriate. Everything is not a nail.

     

    Right, because if I was trying to write a quick little data application it would be much faster and easier to roll my own flat file approach, rather than just interacting with a quick MySQL, SQL Server Express, or other 'lite' DB server. Even mobile devices have built in DB servers. (I know my Windows mobile 5.0 phone does, as did my Pocket PC PDA)

    You are right, what was I thinking?

    Not to mention you can never really be sure when your little personal use app is going to be used by hundreds of people, or scale considerably (I am willing to bet it has happened to most or all of us at some point). I would rather just use the quicker, easier, more scalable solution right from the start. Call me crazy, but I never look forward to rolling my own flat file DB.


  • Considered Harmful

    @dlikhten said:

     Huh, what? I started the desktop search thread? oook.


    I was talking about the previous post in the thread. Your post just happened to be quoted inside of his post. Lighten up.



  • @MasterPlanSoftware said:

    Right, because if I was trying to write a quick little data application it would be much faster and easier to roll my own flat file approach, rather than just interacting with a quick MySQL, SQL Server Express, or other 'lite' DB server. Even mobile devices have built in DB servers. (I know my Windows mobile 5.0 phone does, as did my Pocket PC PDA)

    You are right, what was I thinking?

    Not to mention you can never really be sure when your little personal use app is going to be used by hundreds of people, or scale considerably (I am willing to bet it has happened to most or all of us at some point). I would rather just use the quicker, easier, more scalable solution right from the start. Call me crazy, but I never look forward to rolling my own flat file DB.

     

    erg...sorry... I went off on a tangential rant about over-engineering in general (which I will still stand by), rather than concentrating on this specific case.

    Yes, I'd never do a flat file DB for that Movie DB (one day, I may even get around to starting it - that Excel spreadsheet is getting long), but on the other hand, I wouldn't bother making a massive enterprisey infrastructure and n-tier for a single user application. That was kind of the gist I was coming from.




  • @Quinnum said:

    @MasterPlanSoftware said:

    Right, because if I was trying to write a quick little data application it would be much faster and easier to roll my own flat file approach, rather than just interacting with a quick MySQL, SQL Server Express, or other 'lite' DB server. Even mobile devices have built in DB servers. (I know my Windows mobile 5.0 phone does, as did my Pocket PC PDA)

    You are right, what was I thinking?

    Not to mention you can never really be sure when your little personal use app is going to be used by hundreds of people, or scale considerably (I am willing to bet it has happened to most or all of us at some point). I would rather just use the quicker, easier, more scalable solution right from the start. Call me crazy, but I never look forward to rolling my own flat file DB.

     

    erg...sorry... I went off on a tangential rant about over-engineering in general (which I will still stand by), rather than concentrating on this specific case.

    Yes, I'd never do a flat file DB for that Movie DB (one day, I may even get around to starting it - that Excel spreadsheet is getting long), but on the other hand, I wouldn't bother making a massive enterprisey infrastructure and n-tier for a single user application. That was kind of the gist I was coming from.

     

    Right, and I am only arguing about someone like dlikhten sitting there advocating rolling your own flat file DB instead of just throwing your data into a 'lite' DB server.

    Not like I should be surprised he would be advocating something like that.


  • @medialint said:

    Not sure what the relationship between VB and flat files is. I've got VB happilly chatting with Oracle over here.

     

    It's probably more to do with implicit association than anything else. I remember all the Fisher-Price My First VB Books that used flat-files as a data storage mechanism and threw in things like "Naturally, in a production application, you would use a scalable enterprise words...words...words..."


Log in to reply