Pumped up



  • I have just had a very frustrating week tacked onto the end of a death march project (a wtf in itself, but the classic oversold functionality and unrealistic deadline) so now its time to vent with a posting!

    I am working with an Ethernet connected smart camera that has lots of fancy pattern recognition capability built in to it for extracting and classifying features and is destined to monitor "parts" on a production line at a 40mS rate. The camera comes with its own Windows based development environment that allows for point and click generation of fancy programs that you download to the unit which then performs your specialized image recognition. The units themselves are impressive but do come at the price of a good quality used medium sized small car. In our application after the image classification is performed, we FTP data off the unit and onto a generic computer for historical analysis - which is where this posting starts to come in

    In addition to receiving data from the camera via FTP we also have a need to write control information to the camera before and during a production run. For this the camera company supplies a .Net based SDK that allows you to connect to the camera, read and write data and also subscribe to various events and in general have full access to everything using your favorite CLR language. But over the last week this SDK has turned into the bane of my existence and probably cost me a week of lost time.

    It all started when I noticed that data arriving at my user interface program seemed to stutter when it should have been in lockstep with the processing rate of the camera, even though that rate had been slowed down to 1000mS. Had I screwed up the first WCF link I had every created that was supplying data to the UI? Was my ancient laptop computer too slow? Was it something else I didn't know about? I had already had issues getting the WCF link to work (hint: When using WCF named pipes as an interprocess communications mechanism they have by default an array size limit of 16384 buried deep down in XmlReaderDictionaryQuotas of the NetNamedPipeBinding, which causes all sorts of hilarity when you have an a byte array image of 20kB) so I spent hours pouring over what performance issues they could have. Eventually I started to rule out WCF as the culprit and started looking for other causes. My WCF link was buried in a DLL in a plugin system, so I instrumented the plugin and saw that the stuttering was happening well before the WCF link. Next I went all through my plugin code and tried to find a reason for things screwing up. Eventually I started to see the light and realized that the stuttering was in fact in the data being received by FTP and not in my code, and that it only occurred when I was using the SDK to connect directly to the camera and that my code was faithfully reporting the data as it arrived. With that information I whipped up a test program that just connected to the camera and then watched in horror as the rate of data being received by FTP jumped all over the place (yet seemed to average out to the expected 1000mS). I then emailed tech support with my observations and waited.

    And waited and waited and waited. 2 days later tech support got back to me. And then started the fun conversations of how I could see the problem but they couldn't duplicate it. I refined my test case and wrote minimal programs to cause the issue. I wrote a test program to analyze the rate at which FTP data was received down to the mS. Tech support finally saw the issue when running my code and using my analysis program, but they couldn't see it when running their code. Was it my computer environment, the FTP server being used, the tools used to build the code? I changed them all and conclusively proved that it was either the camera, the SDK or my program - and my program only called the SDK in order to connect to the camera and nothing more. 5 days later I still didn't have a solution, my test code would trigger the problem but the tech support people couldn't trigger it with their own code.

    Finally I took a good hard look at the code the tech support people were using. I traced out all of the functionality and proved that my code was doing exactly the same as their code in terms of SDK calls. Then I sat back and thought long and hard as to what was different between their code and my code - and finally I saw the difference, went D'oh and slapped my forehead! My code was a command line solution, their code was a WinForms solution. By default they were servicing the windows message pump and I was not! The only possible explanation was that by simply connecting to the camera via the SDK and not servicing the windows message pump, something was running amuck in the camera and upsetting the scheduling of the FTP send routines. So time for a new command line test case that serviced the message pump and lo and behold - no more FTP stuttering! I sent that piece of information off to tech support as well as noting that I had to service the windows message pump at least as fast as the camera was triggering - which in my case would be at least every 40mS

    2 days later I finally got a reply back from tech support saying that their SDK does not support non-WinForms programs. Just for yucks I went back over the SDK documentation and noted that there was no obvious mention of this WinForms limitation but I did see that the only example code they supply is WinForms based. If only I had taken that hint earlier, but then again if only I wasn't on a death march project either.



  •  Between the lines I'd guess they weren't really aware of the limitation either, until you pointed it out.

     



  • @OzPeter said:

    I have just had a very frustrating week tacked onto the end of a death march project

    Nasty. When dealing with new technology, you want to do that as early as possible, certainly not at the end of a project, when it can %^&$ up everything and you have no time to figure out an alternative. Because there is always sth like this.

    So at least can you use a winform solution? Or if not, is this all the customer's fault who specifically, against your written warnings, selected this camera/SDK? Or if not, can you at least slap tech support, the project manager and the customer?


  • Considered Harmful

    FTP seems a poor choice for anything real-time.

    It's still not as bad as relying on WINAPI window messages for anything beyond the user interface.



  • @Wrongfellow said:

     Between the lines I'd guess they weren't really aware of the limitation either, until you pointed it out.

     

    Well a couple of weeks ago I was trying to get my stuff running on Win 7. I ran into an issue whereby the camera was failing to send FTP data to the computer. I discovered that it failed when using the MS FTP server, but worked with the FileZilla FTP server. I went down to the level of Wireshark to show them the differences in their protocol responses to the FTP servers. Tech supports reply was that they didn't support Win 7. I'm like .. WTF it a protocol here that is failing .. not a platform. Fortunately we had to regress to XP.



  • @b-redeker said:

    So at least can you use a winform solution? Or if not, is this all the customer's fault who specifically, against your written warnings, selected this camera/SDK? Or if not, can you at least slap tech support, the project manager and the customer?
    My product is based around a windows service so I have to manually serve the windows message pump. I have used this camera previously but I had never needed to connect to it while doing FTP. As for the slapping it will commence with the sales guy (but who pays my invoices) for numerous reasons. The ultimate customer is not at fault - they believed the sales guy that this was going to be easy even though we had never tried the required configuration and the performance specs resulted in requiring a from-the-ground-up-rewrite. Initially I had qualms about even getting data out of the camera fast enough (running at around 25% of Ethernet capacity). But we can still cause the camera to crap out if we FTP too much data in a row. There is no "official" project manager to slap around as it was deemed a 2 person job and I am just a sub-contractor. And even though I am weeks late with my stuff, which gave the other guy heaps of extra time to do his stuff - he has now become the critical path over not completing what I consider to be easy stuff.



  • @joe.edwards said:

    FTP seems a poor choice for anything real-time.

    It's still not as bad as relying on WINAPI window messages for anything beyond the user interface.

    I work with what I have .. **sniff**


  • @joe.edwards said:

    FTP seems a poor choice for anything real-time.

    +1

    @joe.edwards said:

    It's still not as bad as relying on WINAPI window messages for anything beyond the user interface.

    +1

     



  • A bit off-topic, but why do you write mS instead of ms?



  • @Zecc said:

    A bit off-topic, but why do you write mS instead of ms?
    Because I live in an alternate universe where people use S for seconds rather than the SI unit of s



  • I really don't think I'd want to be servicing a pump at all.



  • @OzPeter said:

    By default they were servicing the windows message pump and I was not!

    Did you say it is WCF-based? Than I am not much surprised.

    It has to wait for incoming data somehow. It has two options–event loop or special thread. It probably relies on the former and since the event loop is not running, it is not reading the connection, eventually causing the buffers to fill up and something fails on the other end.

    Note, that you can just run the main loop manually (it's called something like Dispatcher) without pulling in WinForms.



  • I bet thay will put the CLI support into the next paid major version

    That's business !

    Also: sucks to be you ! Doomed projects are never pleasant



  • From what I understand, your program running on the client machine has to run the Windows message pump to get the camera to report data at the expected (correct) intervals?

    That's some WTF.


Log in to reply