Today's episode of Blame the Tool, featuring: Python, FTDI, a library written by niche hardware vendor in 2007, MSVC, and libusb.
One of my colleagues is an undergrad. He is not interested in programming per se, but as an exchange student he took a course in "Python for physics". The result is that he can produce somewhat convoluted iPython notebooks with sometimes questionable software development practices, but the problem gets solved and he never repeats mistakes. This story is not about his code.A few months ago he needed to control some motorised stages from his scripts, while vendor's code was all mostly .NET or LabView. The vendor had also supplied a 300-page PDF describing the protocol that the stages spoke over a virtual serial port. This has been one of the best protocol descriptions I've ever seen and about the only time I could write a bunch of code without even touching the hardware, then plug it in and have the code work on first try. The only problem was actually establishing the communication.
You see, the motorized stages used an FTDI chip. On Windows, FTDI chips can work in two different modes: either the driver provides a COM port for your application to work with as with any other serial port, or there is only the "direct (D2XX) interface" which you can only use with a DLL from FTDI. The motorized stage vendor had apparently chosen the latter option. They also patched the driver and changed the USB PID, so the virtual serial port drivers we got from the official FTDI website refused to work. (At the time we couldn't find any "virtual serial port" FTDI drivers on the motorized stage vendor website, but later I heard a rumour that they exist. Whatever.)
The official D2XX Python module exists, but seems unmaintained. We found PyFtdi but couldn't get it to work. After some frantic and not very systematic searching we found pylibftdi and couldn't get it working either until we realized that the libraries employ libusb
and libusb
on Windows requires special libusb
drivers. Fuck you, Windows, why does it have to be so complicated? Then we found Zadig and finally made some progress. I provided a few examples of how to format device commands to the student, he implemented the rest of it, got the project done in time, and all went well.
There also was talk about controlling our laser from Python. The laser vendor only gave us a DLL written in Delphi circa 2007 and a C++ header for it. I investigated using SWIG to create a Python wrapper for it, but there was no immediate need for it and I had much more pressing stuff to do, so it remained unfinished.
Fast forward to a few days ago, when the student approaches me with a strangely unreadable expression on his face. He wants to "engrave" a certain bitmap on a specially purchased metal plate using the lab laser and motorized stages. He clearly realises that it could be done better for a modest price in a shop using a different kind of laser better suited for engraving; the point is doing it "with his own hands". He remembers that I had some code to control the laser from Python; can he please have it? On his desktop I see a few copies of the same picture, stored in different formats and opened in different programs. The picture is of a smiling female face. Oh, and the deadline is tomorrow morning; he regrets not getting the idea earlier and is ready to spend all night in the lab if he has to, but he needs that code.
I roll my sleeves and install on a virtual machine (he uses Windows; my primary OS is Debian) the same version (for reproducibility, I say to myself) of Anaconda that he uses. python setup.py build
asks me for MSVC 14.0
. I locate the Visual Studio downloads and get told that I have to log in before I can download older MSVC versions. Fuck that! Fortunately, latest Build Tools installer offers MSVC 14.0 among other things, so it turns out I don't have to get yet another stupid account. Unfortunately, it still takes some time to download a few gigabytes of stuff and install them. Some time passes.
After the installation is finished, I try to build the module again. After I get the setup.py
invocation to link against external library right (I actually have almost no real experience with Python) and produce a correct *.lib
file (the one shipped by vendor caused "unresolved external symbol" errors despite being found by the linker), I get this:
1>LINK : fatal error LNK1158: cannot run 'rc.exe'
Fuck you, Microsoft! Why would your link
call rc
which may not be installed? Fuck you, Python! Why would you need a resource compiler to build a damned DLL? (I'm still not sure who's more guilty, so fuck them both.)
Cursory web search tells me that rc.exe
is just plain missing from newer Windows 10 SDKs but could be found in older SDK versions. I make a misguided attempt to install an older Windows 10 SDK which wastes time but does not help at all. Somehow I notice a "Windows XP compatibility something (not recommended)" among the stuff available to install; it is also said to contain the Windows 7 SDK. I copy rc.exe
and rcdll.dll
over into Windows 10 SDK because there is no time to find out how python setup.py
invokes vcvarsall.bat
and have it choose the correct SDK version; the module compiles and even links, so I type:
> python
>>> import lsctrl
ImportError: DLL load failed: The specified module could not be found.
Okay, wait, let me place the DLL in the correct place,
> python
>>> import lsctrl
ImportError: DLL load failed: %1 is not a valid Win32 application
And the magnitude of my own folly was revealed to me in a blinding flash... Of course, my Anaconda installation was 64-bit while the DLL was 32-bit all along. I should have thought of it before.
Should the student grab a copy of 32-bit Anaconda (yet another long download) and install all his modules again? I remembered the exact steps that led us to success with pylibftdi
being not very well defined and probably hard to repeat. Time was ticking, I was not feeling exactly well (I later found out that I had a 37.7℃ fever), I kept making mistakes.
For my other colleague who gets to support our legacy machine I had previously written a C++ library to control the motorized stages. (Officially supported drivers and apps don't work on anything older than Windows 7, but there is a libusb
driver for Windows 2003, so I based the library on libftdi). Again, there had been no immediate interest in it (just a long-term "we will need it in an unspecified amount of time later"), so the library was just a bunch of methods that I had tested on my laptop.
I grab a copy of someone's helpfully provided libftdi build for windows (Building one myself seems like a huge undertaking because of multiple levels of dependencies and probably non-trivial patches required to get it working together. Fuck dependencies!) and throw together a program that uses my library to talk to the motorized stages and vendor's library to talk to the laser. I curse and try different compiler and linker flags and somehow manage to compile everything into a program with two visible different versions of libgcc
(and probably libstdc++
too) in it (one of them is the dependency of my own code and another is required by libftdi
build).
Around this time we notice that USB-RS232 converter that we had planned to use with the laser is busted (it shows an error in the Device Manager no matter what we do), so a separate expedition is mounted to a lab previously belonging to a colleague who died a few years ago because someone remembers him having a working converter. I don't participate in it; I have code to fix.
The code doesn't work, and the only thing libftdi
can tell me is (-4, "usb_open() failed")
. Fuck you, libftdi
, why couldn't you be more specific? I guess it's my fault in trusting so much in cross-compilation that I cannot run code on my machine and I cannot debug the code on the student's machine. I get increasingly desperate and try random stuff that doesn't help until a realisation hits me that the two libraries don't have to share the address space if the universe doesn't want them to.
UNIX way, right? One small program for one small purpose? I throw away all FTDI-related code and only leave calls to vendor laser library: initialize, switch on/off, deinitialize, close port. It compiles and works. I show os.system
to the student, he incorporates it in his own code and we witness the creation of the ablation craters forming the first few lines of the bitmap. Then he notes that the bitmap is mirrored and gets back to fixing his code. I wish him luck and shuffle over to home where I take my temperature and get a nasty surprise.
There is no real ending; we didn't discuss that ever again.