I don't even know...
-
At the beginning of December, we installed a new storage server for a client, to be used as a PACS server for medical images of all types. They have their own digital x-ray, ultrasound and CT scanner, etc. and a PACS server allows all of the doctors access to a central repository.
Our only responsibility was to build the server, install the OS, etc. The medical imaging company that sold them the upgrade to their CT and X-Ray said they would install the software, setup the clients, etc. OK, fine. The only thing that they told us was that their DICOM server software did not play well with dedup, so that had to be turned off. Odd, that should not come in to consideration, and should be transparent to any application, but whatever... They asked about backups, I told them we would take care of that.
That was a month ago, and the server was a 12TB (six 4TB drives in RAID10) behemoth meant to last them for a while. I get a call this morning that the digital x-ray machine cannot save to the PACS server. No space. What...the...fuck? A month ago their storage footprint on the server that was replaced was under 1TB. I am unable to remotely login to the PACS server. Hmmmm, if the disks are sitting there at 100% capacity the server would certainly be thrashing. So, I check the backup target server and it is showing 10+TB of backups...but 90%+ dedup. Something is odd.
I go to the client's location and check the PACS server. It allowed me to login while sitting at the machine, thankfully. It appears that they scheduled their own database backup on the server, to the server, and the backup...included all prior backups as part of each concurrent backup. Like an Ouroboros of backups or something. Or maybe like a backup of Tribbles. I suck at amusing metaphors today.
So, multiple WTFs. They backed up a machine, to itself. Not entirely a WTF, as long as that backup gets sent somewhere else, etc. Dumping your backups to the exact same location as the live data, so that it gets included as part of the next backup...that is TRWTF.
-
-
it's also not an effective backup at all.
Yeah, but it depends on the purpose of said backup. Against failure, it is rubbish. To revert back to a point in the past, it is serviceable.
But yeah, overall it is a rubbish idea...
-
it's also not an effective backup at all.
Not entirely. It could be used to restore accidentally deleted stuff.
-
Not entirely.
granted, but it is all but useless. and there are better ways to do deleted file recovery.
-
It could be used to restore accidentally deleted stuff.
Yep, or a quick recovery of a database that shit itself beyond repair, etc.
For recovery from storage failure, it would be useless. For all the other reasons you might need a backup, it would be perfectly serviceable. You just need to make sure you have not set up a recursive backup. :)
-
Like an Ouroboros of backups or something. Or maybe like a backup of Tribbles. I suck at amusing metaphors today.
No no, that first one was pretty good.
/me pretends to wonder what made you choose the word Ouroboros.
-
it's also not an effective backup at all.
It might be useful as part of a backup. Do a hot DB backup to a directory, have another process come along later and backup the DB backup somewhere else. I've seen that used quite a bit.
-
pretends to wonder what made you choose the word Ouroboros.
In retrospect, Matroska dolls would have been a more apt metaphor...
-
Backupception.
-
"Yo, dawg, I heard you like backups..."
-
"Yo, dawg, I heard you like backups..."
I thought of that one also...
The rabbit hole goes deeper. Someone remind me later to complete the story. Trying to get a toddler to bed at the moment...
-
Someone remind me later to complete the story.
Complete the story later!
I'm here all week...
-
And?
-
-
Thanks for the bumps fellas, I fell asleep in the rugrat's room last night and never made it back to my computer.
So, after we get the Matroska dolls unnested, and peel back all of the layers, and get the proper files at the root of the folder where they should be, I do a search of all images with no search parameters. I want all of the studies. Then I sort those by which had been uploaded. There are tons. 1300 some that never made it on to the PACS server. Most of those are from before the new system was put in to place, which means that the company who sold the new system to my client never verified their import, etc.
Fine, cue those in batches for upload. Almost all complete without issue, but there are 93 from the last month that just error out. Check the error logs (which are not easy to find) and get this helpful error message:
Upload to storage server failed with reason: Storage server upload failed.
That is not verbatim, but you get the jist of it. I check the PACS server and get mostly the same error message. Nothing meaningful to tell me why it failed. Hmmmmm, what separates these 93 radiographs from the others? What is different?
Well, not much. The images appear to be fine, so they are not rejected because of corruption. All studies are basically the same size as they are essentially TIFFs in a proprietary wrapper, so it is not a size issue. The only thing I find is that these 93 have slightly longer comments on the images. I truncate the comments on one of the studies, click on "Upload to DICOM" and it goes right through. Do the same to a second one, fails. Shorten it a little bit more, success.
As it uses a proprietary form of database, I am guessing they have no BLOB or similar type format and it is of a fixed length. I did not count, or poke around, but ~256 characters.
So, pick your next WTF:
- Text input field does not alert you as to how many characters you can input before getting an error (they are made to work together, so it was a known quantity)
- Error logs give such a shitty error message for something that should have been known
- Database not having an open-ended text field such as BLOB
- Text input field should not even let you put in too much text for an upload
- No visual indicator on GUI that an upload has even failed. It only shows back in an admin section that shows the file transfer queue
Probably some others that I am missing...
Addendum: How fitting that I get this error message from Discourse when I click Submit...
-
Text input field does not alert you as to how many characters you can input before getting an error (they are made to work together, so it was a known quantity)
But was the thing created with a previous version of the software? That was my impression of events, but maybe I'm wrong. Not that you'd expect them to shorten data lengths, or maybe the current version validates the input?
-
But was the thing created with a previous version of the software?
Current version of client-side software. It was only installed a month ago, and both of them were installed at the same time. Client went on a buying spree and replaced their CT, digital x-ray and ultrasound machines all in one fell swoop and from the same company, and essentially the same brand. They should have worked together seamlessly. The viewing software that the doctors use is a different brand, etc. Had the error been there, it would have been understandable.
Also, DICOM/PACS/(whatever other acronyms that they throw into this quagmire) are all standardized. The spec should be known.
-
Addendum: How fitting that I get this error message from Discourse when I click Submit...
I think this means discourse does not enjoy your post and values you personally as a zero, and is disinclined to help you distribute your stories. Seems rather judgmental to me, but we're all about quality posts here, obviously.
-
I think this means discourse does not enjoy your post and values you personally as a zero
That's not what it means at all. I mean, you're not wrong in your supposition of Discourse's opinion of him. It just means that maybe the post saved and maybe it didn't. I've seen both outcomes.
but we're all about quality posts here, obviously.
Maybe in this thread...
-
Maybe in this thread...
Not even in this thread. I know that any thread I start has a significantly less than zero chance of devolving into a shit storm.
-
I don't suppose you could name and shame the vendor could you?
Most people would be surprised by what kinds of workarounds/hacks doctors/nurses/techs use when dealing with Healthcare software and abusing "comments" fields is definitely one of them. I do hope someone is going to tell the end users about that comment field limitation right?
In my experience Healthcare standards are all structural. IE: HL7. Two systems can "read" HL7, but can barf because of CR vs LF issues.
-
Not even in this thread. I know that any thread I start has a significantly less than zero chance of devolving into a shit storm.
I are confuzzled. Even leaving aside what P < 0 might mean, this would make more sense if you said "chance of not devolving into a shit storm." I.e., that it is certain to devolve.
-
I don't suppose you could name and shame the vendor could you?
Hmmmmmm, not exactly so. I will say they are a real gem though. Infer from that what you wish. ;-)
-
make more sense if you said "chance of not devolving into a shit storm." I.e., that it is certain to devolve.
Started typing it one way, went back and tried to say it another. Ended up with a hybrid between the two. Also, I am on mobile due to our Internet being down (maybe our internets are frozen? Once they thaw, my packets will flow again. The internets have literal pipes, right?).
-
-
Yeah, went back and re-read it, liked the original way better.
-
Healthcare standards are all structural. IE: HL7. Two systems can "read" HL7, but can barf because of CR vs LF issues.
Right on! Both sides will technical confirm that they confirm to the standard HL7 reading/writing messages and still the shit fly around because someone slightl abused some date field. Eg putting in 'now' in stead of the actual admission date, because hey you should be reading admission date if you want to know that ...
-
It means Blakeyrat's Law wins again.
-
What is @blakeyrat's law, and what does it win?
-
What is @blakeyrat's law, and what does it win?
I think he's referring to the assertion that any discussion conducted using Discourse will eventually wind up discussing (the bugs in) Discourse, itself.
-
I think he's referring to the assertion that any discussion conducted using Discourse will eventually wind up discussing (the bugs in) Discourse, itself.
I thought it was the assertion that @blakeyrant really digs dwarf porn, preferably with amputees?
-
Seems reasonable. We should have a sweepstake on how long we think the law will continue to hold...
I'm calling forever...
-
@blakeyrant really digs dwarf porn, preferably with amputees?
Why does he prefer to watch with amputees?
-
-
Maybe because they can't run away?
-
Depends on the amputation, but I dig your thinking.
-
Like an Ouroboros of backups or something. Or maybe like a backup of Tribbles. I suck at amusing metaphors today.
What about... exponential backup?
-
No cursing of the toaster that I know popped up for you? I am disappoint.
-
Oh c'mon, did you really have to jinx it?
I think he's referring to the assertion that any discussion conducted using Discourse will eventually wind up discussing (the bugs in) Discourse, itself.
-
-
