Database Disaster
-
I've got a buddy (heretofore known as "John") that took a job with a data warehouse of sorts. Now this data warehouse used an obscure, relatively specialized proprietary database system to index tens of thousands of paper-based documents. This database system (which was either designed in the Windows 3.1 or late DOS age) had to be manually reindexed via a relatively complex process on a nearly daily basis. This wasn't a problem for John, as the reindexing was Someone Else's Job, but occasionally the process would fail and the database would be down for a few hours or a couple of days while The Vendor and Someone Else put it back together -- often resorting to restoring the manual backup that was created immediately before the reindexing process. One day, John happened to be in Someone Else's office before they started reindexing and discovered something that might have contributed to the failures that they were experiencing. Someone Else would occasionally fiddle around with the data to edit or (possibly) delete records -- using a special DOS-based text editor! (The special editor had to be used because the "end-of-record" marker was a DOS EOF character...)
This discovery led John to a couple more interesting discoveries -- data inside each field had a very strict upper character limit (which the manual editing was very possibly violating) and database fields were non-unique. In some respect, he supposed it made sense... If the number of characters per field had a hard limit, just make more fields with the same name! If the software allows it, it MUST be okay... Anyway, all of this was still primarily Someone Else's problem -- until the push came to take the database online...
The Vendor wanted very much to write an application to sit between the data and the website to bridge the then-ancient database system with the modern world, but he was proposing a $30,000 price tag and an estimated completion time of a month. John warned his boss repeatedly that it wasn't the best option available, as the developer had made an ActiveX control that John could use in a CGI program to display the search results. The boss wouldn't listen until John, on the very day that the boss was set to accept The Vendor's terms, showed him a completely functional proof of concept.
Eventually John took up the task of converting the data to an SQL-based database, which was no small feat in itself (combine all the same-named fields into one field, make sure that the record identifiers were unique without substantially renumbering, etc), but every time he got a phone call from another company as a reference for The Vendor's product, he was always very careful to say, "I'm sure it's great -- if you use it the way it's intended to be used..."
So I don't have a lot of knowledge about or experience with ancient database systems -- is this normal? Or is the WTF the fact that they used the database long after their needs outgrew it?
-
The WTF is the fact that they used a text editor to modify the database. This could have possibly (and seems to in his case) corrupted the entire database. Although people should try their best to make current, the fact that companies have ancient DB systems is not a WTF. The WTF is the way that some people got "creative" when interfacing into that db, and the workarounds they employed (such as making two fields with the same name, instead of making fields like FirstName1 and FirstName2). That might not be a good example, but the fact of the matter is that people should attempt (ha ha) to use good database practices that helps reliably maintain the data.
-
@studean said:
So I don't have a lot of knowledge about or experience with ancient database systems -- is this normal? Or is the WTF the fact that they used the database long after their needs outgrew it?
You mention that this is probably from the "Win3.1" days. A friend of mine recently inherited a database which is known to predate Win3.1, so I guess these ancient databases aren't that uncommon. Unfortunately, there is no documentation on the file format, but he has apparently managed to hear tell of some guy who no longer works in the state who once dealt with the internals of that database app who may be able to help him reverse engineer the files and stuff the data in something newer...
-
Could the database have been "PROGRESS". That disasterbase is a WTF in and of itself.
-
Ok, I have to know what you know about Progress DB, since our company is embarking on a new ERP system (I use those terms as loosely as possible) using the Progress DB. We have already figured out this is going to be a huge source of WTFs... I can list a couple dozen already, but I would like to know more about your experiences....
Enlighten me?
PS: Love the term disasterbase, we are going to use that instead of Progress from now on....
