Architecture Woes



  • I've promised my comrades in the channel an explanation of our fail architecture.  My client (let's call them STEVE) manufactures an important piece of a larger piece of machinery for the US and other governments.  I hope I've anonymized this enough.

    On to the process. 

    We download files from the complete machinery manufacturer (let's call them ROBERT).  It goes like this.  They have a SOAP service which we poll so we know when we have a message indicating that a file is available to download.  When we get that message, we go to an sftp site and download the file that's listed in that message.  Once we get the file, we immediately store it on the NAS and send a message to our next process via MQ.  

    The next process (The Load Process) receives the MQ message, pulls the file off the NAS and inserts it into a datablob field in an Oracle database.  Once it's done that, it deletes it from the NAS.

    The third process pulls the file out of the datablob, extracts the data, exports it to xml and throws it at the data warehouse to be loaded there.  It leaves it in the database.

    The final process pulls the file out of the datablob and puts it on another NAS for long-term (50 year) archival.

    Additionally, we just got a new process for interacting with ROBERT.  Sometimes the messages (which AFAIK are just a list of files to download) may be too big to download with SOAP(!) so the message will indicate where I can go on the FTP server to download the full message.



  • @belgariontheking said:

    I hope I've anonymized this enough.

    Don't worry, nobody at Raytheon will know a thing.

     

    Also, congrats on popping my 3k post cherry! 



  • Those asshats at Raytheon wanted to hire me, they wanted me to work out in the middle of the desert.  Wasn't going to happen.  

     

    @morbiuswilters said:

    Also, congrats on popping my 3k post cherry! 

     

    Congrat!  What do you win?



  • @amischiefr said:

    @morbiuswilters said:

    Also, congrats on popping my 3k post cherry! 

     

    Congrat!  What do you win?

    A copy of "101 uses of The Shocker on a dead cat"



  • Ooooh. A Rube Goldberg contraption. It reminds me of that one job which involved SAP -> tab-delimited flat file -> DB -> tab-delimited flat file -> another DB -> another tab-delimited flat file -> SAP (or something like that.)



  •  @danixdefcon5 said:

    Ooooh. A Rube Goldberg contraption. It reminds me of that one job which involved SAP -> tab-delimited flat file -> DB -> tab-delimited flat file -> another DB -> another tab-delimited flat file -> SAP (or something like that.)

     It could have been worse.  It could have been SAP -> XML file -> DB -> XML file -> another DB -> another XML file -> SAP :)  Of course, you may need Java (or more Java) to fill the cracks.

    I don't know how WTF-worthy this is, but:  In high school I did a job shadow with a well-known mortgage company.  If I recall correctly, they relayed information between their Java-based web application (under development) and an IBM COBOL database server via a third party company because (they said) Oracle or MySQL couldn't handle the volume of data they needed to handle.  Thus, the communication chain was:  Java application on Windows <- XML -> Third party server <- XML -> IBM database server.  The project took over one million man hours (and counting) and login response times are around 60 seconds.

    Hopefully they'll be able to improve their response times after they get bailed out.



  •  Man, you think your company is a WTF?  Well mine just stopped offering free coffee in the break room, yet they still offer tea!  The reasoning?  "Tea is cheaper."  Yeah, you know why tea is cheaper?  Because tea is a sucky coffee wannabe and nobody likes it.  It's really cheap when you only ever have to buy 1 box of tea bags.



  • @bstorer said:

     Man, you think your company is a WTF?  Well mine just stopped offering free coffee in the break room, yet they still offer tea!  The reasoning?  "Tea is cheaper."  Yeah, you know why tea is cheaper?  Because tea is a sucky coffee wannabe and nobody likes it.  It's really cheap when you only ever have to buy 1 box of tea bags.

     

    Coffee is not good for you. Cofee will give you irritable Bowel Syndrome as well as heartburn. There are studies that link the consumption of coffee for prolongued time with osteoporosis.

    There is also the shakiness, concentration problems and an elevated chance of a heart attack. There is also the caffeine crash. It transforms you into a junkie in the morning drinking a couple cups of coffee and after the energy drop an hour or so later you are looking for sugar or more coffee to try to get going again. 

    One of the big problems with many people is that they replace a good breakfast with a couple of cups of coffee and this means that in addition to having the elevated heart rate they are also living with a far less healthy diet than the non caffeinated people around them.

    Also it gives you bad breath. I hate people in the elevator with cofee breath.

    All in all, I think your company did a good deed. You will thank them later.  

     



  • @fatdog said:

    COFFEE BAD!
    I know you're trolling, but it warrants mentioning that none of what you said resolves the key issue: tea sucks.



  • @bstorer said:

    tea is a sucky coffee wannabe
    No it's not.  Tea is superior to coffee in every way:  tastes better, doesn't stain as severely, is lower in harmful stimulants.  All this and it still gives you a pick-me up.



  • @belgariontheking said:

    @bstorer said:

    tea is a sucky coffee wannabe
    No it's not.  Tea is superior to coffee in every way:  tastes better, doesn't stain as severely, is lower in harmful stimulants.  All this and it still gives you a pick-me up.

     

    Agreed. However, [b]tea bags[/b] suck indeed. 



  • @belgariontheking said:

    tastes better

    No.

     

    @belgariontheking said:

    doesn't stain as severely

    Some of us can drink liquids without spilling them everywhere.

     

    @belgariontheking said:

    is lower in harmful stimulants

    Wait, weren't you listing things the benefits of tea and not its horrible, sucky downsides?  lern2pros, lern2!cons



  •  Six (now seven) posts about coffee and tea, nice!



  •  That was (is?) a beautiful thread derail.

    Back to the point, this contraption has a serious lack of CSV. Excel-style CSV. Oh, and COM interfaces. (does anyone use these anymore?)



  • @Ilya Ehrenburg said:

    @belgariontheking said:

    @bstorer said:

    tea is a sucky coffee wannabe
    No it's not.  Tea is superior to coffee in every way:  tastes better, doesn't stain as severely, is lower in harmful stimulants.  All this and it still gives you a pick-me up.

     

    Agreed. However, tea bags suck indeed. 

    However, you can suck tea bags.

     



  • @danixdefcon5 said:

    tab-delimited flat file
     

    Surely there must be more to that


  • :belt_onion:

    Nice implementation of the publisher-subscriber pattern!



  • @dstozek said:

     That was (is?) a beautiful thread derail.

    Back to the point, this contraption has a serious lack of CSV. Excel-style CSV. Oh, and COM interfaces. (does anyone use these anymore?)

     Sadly, yes. Web Input -> Oracle Database -> Custom Flat File(Format1) -> XML -> SFTP Offsite -> XML -> Custom Flat File(Format1) -> Oracle Database -> COM interface -> Custom Flat File(Format2) + SQL Database -> Printer.

    I've combined a few steps for clarity, I hope it doesn't ruin the beauty of the design.



  • You know what we do? We convert CSV to XML because it's easier to process in SQL Server.

    (I was the one who wrote the converter...)



  • @SlyEcho said:

    You know what we do? We convert CSV to XML because it's easier to process in SQL Server.

    (I was the one who wrote the converter...)

    For us, it's not just CSV to XML, it's CSV AND XML and even code together, so a file looks like this:

    12345,"wtf","FileNotFound","<wtfxml>
      <function name=""getPaula"">
        <parameters></parameters>
        <code><![CDATA[
          //here is more XML within code within XML
          db_query(""SELECT FileNotFound FROM wtf"", ""<query db='main'/>"");
          return ""Brillant!"";
        ]]></code>
      </function>
    </wtfxml>"
    

    The repeated double quotes are the CSV escape sequence for quotes.



  • @Ilya Ehrenburg said:

    Agreed. However, tea bags suck indeed. 
     

     You've obviously never been teabagged.

     

     

     

    Sorry to make my first post here dirty. Couldn't help it.



  •  Didn't even know the verb. So had I been teabagged before, I wouldn't have known.



  • @dogalmity said:

     

    Sorry to make my first post here dirty. Couldn't help it.

    Don't apologize.  If you make every post dirty, you'll soon be king of this forum.


  • @Ilya Ehrenburg said:

    Didn't even know the verb. So had I been teabagged before, I wouldn't have known.
    Wait, so what did you mean when you said that tea bags suck?  They don't literally suck, and they're not difficult to operate.



  • @belgariontheking said:

    @Ilya Ehrenburg said:

    Didn't even know the verb. So had I been teabagged before, I wouldn't have known.
    Wait, so what did you mean when you said that tea bags suck?  They don't literally suck, and they're not difficult to operate.

    The quality of the stuff they put into tea bags is usually poor, so they suck in the sense of not producing good tea. Try brewing tea from real leaves some time, you'll notice the difference.

  • :belt_onion:

    @bstorer said:

    @dogalmity said:


    Sorry to make my first post here dirty. Couldn't help it.

    Don't apologize.  If you make every post dirty, you'll soon be king of this forum.
    I usually post dirty as well, but take long hot showers afterwards to wash away the pain


  • @bstorer said:

    If you make every post dirty, you'll soon be king of this forum.

    That would assume people enjoy morbiuswilter's posts...



  •  More than yours, certainly. Isn't it boring for you, too, to repeat "morbiuswilters is a stupid troll" up to 50 times in every thread?


  • :belt_onion:

    @Ilya Ehrenburg said:

    @belgariontheking said:

    @Ilya Ehrenburg said:

    Didn't even know the verb. So had I been teabagged before, I wouldn't have known.
    Wait, so what did you mean when you said that tea bags suck?  They don't literally suck, and they're not difficult to operate.

    The quality of the stuff they put into tea bags is usually poor, so they suck in the sense of not producing good tea. Try brewing tea from real leaves some time, you'll notice the difference.
    And tea bags aren't really bags, are they? You can't put anything in and you can't take anything out. That's the least you expect from a bag!

  • Discourse touched me in a no-no place

    @bjolling said:

    And tea bags aren't really bags, are they? You can't put anything in and you can't take anything out.
     

    http://www.herbalremedies.com/muslin-bag.html

    http://www.teazonline.com/servlet/the-174/Reusable-cotton-tea-bag/Detail



  • @fatdog said:

    There is also the shakiness, concentration problems and an elevated chance of a heart attack. There is also the caffeine crash. It transforms you into a junkie in the morning drinking a couple cups of coffee and after the energy drop an hour or so later you are looking for sugar or more coffee to try to get going again.

    Most tea also contains caffeine.  This is also a straw man, as it is not necessary to abuse the coffee; you can just have one or two cups of coffee a day or you can have decaf.


    One of the big problems with many people is that they replace a good breakfast with a couple of cups of coffee and this means that in addition to having the elevated heart rate they are also living with a far less healthy diet than the non caffeinated people around them.

    Red herring.  Skipping breakfast is not good.  Having only a cup of tea for "breakfast" is also not healthy.


    Also it gives you bad breath. I hate people in the elevator with cofee breath.

    Well, you got me there.


    All in all, I think your company did a good deed. You will thank them later.  

     

    No, they're probably just cheap bastards who cut back his health coverage to compensate. 


Log in to reply