A short story about packets



  • At my company, we make hardware for analyzing network traffic. There's some really neat use of FPGA-accelerated packet matching, which interfaces with some beautifully written parallel-processing software, crafted with deep understanding of CPU cache coherency and OS kernel buffer management. So I'm told, anyway, by the "Core Tech" team who write the stuff. They truly are the greatest software engineers we've ever employed - more highly qualified, better paid, and with benefits including visits from a company-employed massage therapis

    You get the picture, anyw

    So when we got the opportunity to interface with their system, we were thrilled. Finally, our lowly Java-based mediation system would get some glory, albeit reflected from the Core Tech t

    I actually got to meet the Lead Engineer - an imposing fellow, with a goatee beard and shiny bald pate, which he covered with a trilby hat. He placed The ICD in my hands: "You will conform to this. No changes are possible, because we've already shipped this to our customers." Not a problem, of course. The interface seemed clear enough, with messages defined as Protoc

    ol Buffers, with some surrounding framing records. Some messages carried XML definitions, but these too were clearly defined with a precise XML schema.

    A few days later, my tech lead wandered over, looking puzzled. "I'm sure I'm doing this right," he said, "but the probe keeps dropping the connection." Sure enough, after every message: "Connection reset by peer.

    I put in a call to the man with the trilby, who snorted down the phone. "Did you even read the ICD? I know you guys have your agile processes, but in my team, we write precise specifications for you to follow." Feeling chastised, I paged through the specification, and sure enough, on page 37, footnote 2, it said:

    (2) one message per connection

    My tech lead and agreed this didn't seem very efficient, but then, mostly we used XML and RESTful web services, so who we

    re we to criticise.

     

    A few days later, my developer hit another snag. Under testing, the system seemed a little erratic; depending on exactly what was sent, the probe sometimes seemed to ignore our messages. Not wanting to interrupt the Core Tech team again, we debugged for a day or so, and then in desparation sent a Wireshark capture over to the Lead Engineer. A matter of minutes later, he shot back:

    "RTFM. P18."

    Chastened, we turned to page 18, and again, in paragraph 3, it said: "Messages must not span packets."

    We looked at the PCAP file, and lo and behold, we had been stupid enough to allow Java to span some messages across multiple packets. My lead developer isn't sure how to fix this one, so I suggest he leaves it with me, and moves on to finish implementing the specificat

     

    A few hours later, while I'm googling the correct use of "writeBytes()", my developer is back again. Wordlessly, he shows me a packet capture, where the probe is sending him truncated XML. He looks visibly upset when I reach for the phone - we must be ruining our reputation as competent engineers. So we go back to the spec, combing through it line by line. Then we see it:

    "No packet shall exceed 2048 bytes."

    We decide to call it a day - neither of us wants to call the Lead Engineer.

    The next day, we try in vain to find an alternative solution, but to no avail. We really need that XML. So we try the path of least conflict: We raise some JIRA bugs.

     

    Two weeks later, they're closed "Fixed". We get the new software and ICD. The change log reads:

    "Updated for the GXS team, who can't follow a simple specification."



  • Not to sound like a dick, but it seems like you checked every resource you had during debugging EXCEPT the packet given to you by these devs. They're being a little snotty about it, but if the other developers in your company are doing the same thing as you (constantly pestering them about problems explained in the documentation in plain English) I think I'd cut them slack for being sick of answering the same damn questions over and over again.


  • Java Dev

     One wonders - why are they even using TCP if they're not supporting 'messages spanning multiple packets'? Sounds like they should've put it on UDP to begin with.



  • @PleegWat said:

    One wonders - why are they even using TCP
    Guarantee of message delivery?

    (or detection of non-delivery)



  • Did you leave out letters in your post as some sort of meta-joke on how their hardware was processing packets? Or what's going on there?



  • Perhaps I'm misunderstanding, but it sounds like this software layer adds more complexity than it solves.



  • @blakeyrat said:

    Did you leave out letters in your post as some sort of meta-joke on how their hardware was processing packets?

    Good spot.

    So, in this (moderately fictionalized account - there wasn't really a bloke in a trilby, or such a detailed spec), we encountered three WTFs:

    1. the software only accepts one message per connection, for no obvious reason other than poor design
    2. it doesn't correctly handle messages split across TCP packets - which is considered a bug, for any reasonably written software
    3. it truncates its own response messages - probably because the developer is incorrectly using a statically allocated buffer

    Coming from software made by a team whose whole existence is dedicated to handling complex network traffic, the sheer lameness of this mediation interface (not the primary packet capture interface) took us rather by surprise. Perhaps they got an intern to write it.



  • BINNED

    @asquithea said:

    @blakeyrat said:

    Did you leave out letters in your post as some sort of meta-joke on how their hardware was processing packets?

    Good spot.

    So, in this (moderately fictionalized account - there wasn't really a bloke in a trilby, or such a detailed spec), we encountered three WTFs:

    Did you apply for a DailyWTF writer position? It looks like you'd fit right in.



  • @PedanticCurmudgeon said:

    @asquithea said:

    @blakeyrat said:

    Did you leave out letters in your post as some sort of meta-joke on how their hardware was processing packets?

    Good spot.

    So, in this (moderately fictionalized account - there wasn't really a bloke in a trilby, or such a detailed spec), we encountered three WTFs:

    Did you apply for a DailyWTF writer position? It looks like you'd fit right in.
    Dang, that's exactly what I was going to say.



  • @asquithea said:

    @blakeyrat said:
    Did you leave out letters in your post as some sort of meta-joke on how their hardware was processing packets?
    Good spot.

    Wasn't really much of a spot, more of a "are you really doing something that lame?"

    But eh. Yeah. Good spot. Yeah.

    (Don't get me wrong, the WTF was good and well-written otherwise, the "cutting off characters" joke was just ... skip it next time.)



  • @asquithea said:


    2. it doesn't correctly handle messages split across TCP packets -
    2.           - which is considered a bug, for any reasonably written software
    3. it truncates its own response messages - probably because the developer is incorrectly using a statical

     

    sounds like the same constraint - memory - this is not a WTF in fpga designs 

     



  • @Helix said:

    sounds like the same constraint - memory - this is not a WTF in fpga designs
     

    Not really - the FPGA stuff is accompanied by a conventional linux server to run the rest of the software, which is written in C++.

     


     



  • @asquithea said:

  • it doesn't correctly handle messages split across TCP packets - which is considered a bug, for any reasonably written software
  • That make no sense at all. TCP is a stream protocol. It does not itself have any concept of packages.

    TCP/IP (TCP over IP) which is what I guess you are using does have ip packages, but there is in general no way to control the size and number of ip packes your tcp stack will generate when sending x bytes.

    So if they really managed to implement a tcp stack which can only handle 1 ip package per connection in total then that is the true wft. And the fact that you can communicate with their system at all is pure luck (Or rather due to a tcp stack optimization where your tcp stack will try to use as few packages as possible.But some day it will split that 1700 bytes.  

     


  • Java Dev

    That's why I wondered why they don't use UDP - If they managed to build a TCP/IP interface that can't span messages accross packets, they must have written their own TCP/IP stack. And somehow I doubt they're doing any of the traffic control stuff on that.



  • Bog standard TCP/IP stack, and as mt@ noted, you don't get any real control over how your data arrives (unless you're interfacing directly with the stack, through something like a PCAP driver).

    But network functions almost invariably return whatever data is available up to a specified limit, and need to be called in a loop. This makes message assembly a bit more painful for the recipient, so if your system mostly receives short messages, it's tempting (but incorrect) to assume you were given the entire message in one go:

    Explanation here: http://stackoverflow.com/questions/9275762/how-c-recv-function-acts-at-data-receving-could-it-receive-a-partial-packe

  • BINNED

    @Sutherlands said:

    @PedanticCurmudgeon said:
    Did you apply for a DailyWTF writer position? It looks like you'd fit right in.
    Dang, that's exactly what I was going to say.

    No worries, it looks like OP hasn't figured out yet that my post wasn't a compliment. I'll let you break the news to him/her.



  • @asquithea said:

  • it doesn't correctly handle messages split across TCP packets - which is considered a bug, for any reasonably written software
  • it truncates its own response messages - probably because the developer is incorrectly using a statically allocated buffer
  • Was something like 1460 bytes anonimised to 2048 bytes? So that responses won't generate more than one packet?



  • @Zemm said:

    Was something like 1460 bytes anonimised to 2048 bytes? So that responses won't generate more than one packet?
     

    I don't think so. The single-packet issue was on receipt, whereas we saw the truncation on the response. I think the programmer simply statically allocated a fixed size buffer to assemble responses into, without bothering to check whether the responses might sometimes be too big, and then called send(...)

     


Log in to reply