The MythTV protocol
-
For those who don't know of it, MythTV is an open source media center software. Unfortunately, it has all the downsides of open source with none of the upsides.
I've been working on a library to connect to a Myth backend, since their existing library relies extensivly on Qt.
Myth commands being sent to the backend have space-delimited strings (mostly), but commands sent from the backend are delimited with "[]:[]". So, this is a common interchange:sent >16 QUERY_FREE_SPACE
Of course, although it appears to be four values, that's actually only two - 64-bit integers are encoded as two 32-bit integers for some reason, which is bizarre, because they're sent as strings anyway! Some responses may contain 40 or more fields. These are accessed using numerical indexes only, so whenever a new field is to be added it must be put at the end AND the protocol version incremented. It's not unusual for the protocol version to be incremented 5-10 times in between releases.
response>34 0[]:[]196014080[]:[]0[]:[]29351936
But wait, it gets better. Commands involving transferring of files are sent to the server with some arguments delimited by spaces, and others by "[]:[]". For example, to request a file, you might send:43 ANN FileTransfer Desktop[]:[]/some_file.png
Of course, there's no documentation at all - I've been reverse-engineering the protocol from the server's log output and the source code.
Now, what if you want to get a file?
First, you must send the version of the library you are using:21 MYTH_PROTO_VERSION 42
If you send the wrong protocol version, the server will send the expected version abort the connection. I get around this by sending protocol 0, then re-connecting with whatever the server sent.
Next, ANNounce your intention to get a file:43 ANN FileTransfer Desktop[]:[]/some_file.png
You will get a string in response along these lines:25 OK[]:[]20[]:[]0[]:[]13987
This means that your transfer id is 20 and the file is 13987 bytes long.
You must re-send the ANNounce message to be able to read from the server:21 ANN Monitor Desktop 0
Finally, you can begin reading data (say, 42 bytes worth):46 QUERY_FILETRANSFER 20[]:[]REQUEST_BLOCK[]:[]42
At this point, with everything else in the protocol being performed with strings, you might think that perhaps the data will be encoded into hex or base64 or something. Nope, straight bytes across the wire, with no length prefix (no way of knowing how long the data will be). After the data has been sent, the server will tell you how much was sent - gee, thanks, that would have been nice to know before I had to allocate a buffer to hold it all.
The worst part is that the backend seems to randomly segfault on valid input. I can't run it in a debugger, because in addition to being the coordinator of the system it handles recording video data (and a debugger slows it down too much to do that).
It's quite a brillant system.
-
@bobday said:
The worst part is that the backend seems to randomly segfault on valid input. I can't run it in a debugger, because in addition to being the coordinator of the system it handles recording video data (and a debugger slows it down too much to do that).
Couldn't you just let it dump core, then run the debugger on the dump? Not as flexible as debugging the live process, but better than nothing.
-
@bobday said:
Next, ANNounce your intention to get a file:43 ANN FileTransfer Desktop[]:[]/some_file.png
You will get a string in response along these lines:25 OK[]:[]20[]:[]0[]:[]13987
This means that your transfer id is 20 and the file is 13987 bytes long.
You must re-send the ANNounce message to be able to read from the server:21 ANN Monitor Desktop 0
Finally, you can begin reading data (say, 42 bytes worth):46 QUERY_FILETRANSFER 20[]:[]REQUEST_BLOCK[]:[]42
At this point, with everything else in the protocol being performed with strings, you might think that perhaps the data will be encoded into hex or base64 or something. Nope, straight bytes across the wire, with no length prefix (no way of knowing how long the data will be). After the data has been sent, the server will tell you how much was sent - gee, thanks, that would have been nice to know before I had to allocate a buffer to hold it all.
Didn't you just ask it for 42 bytes of a 13987 byte transfer so that should give you enough to work out your buffer size?
Probably the size after transfer is a check so that you can compare how many it sent with how many it should have sent?
Seems pretty cumbersome but no worse than commercial systems I've worked with. At least you have access to the source code, whereas with badly documented commercial products it is just a matter of trial and error till you figure out what the documentation could have made clear in the first place...
-
@bobday said:
Of course, there's no documentation at all - I've been reverse-engineering the protocol from the server's log output and the source code.
Have you searched for any documentation?
How about this:
-
@bobday said:
Of course, although it appears to be four values, that's actually only two - 64-bit integers are encoded as two 32-bit integers for some reason, which is bizarre, because they're sent as strings anyway!
Quick test then: how do you convert a 64-bit integer to a string in C?
If you said printf("%lu", x) or something along those lines, and you're using i386, then you're out of luck. long int is only 32 bits on that platform on all the current OSes. And therein lies the problem. GNU C provides long long int, but that isn't portable. C99 provides int64_t, but doesn't provide a printf tag for handling 64-bit values. The fastest, most portable way to transmit a 64-bit integer as a string is to transmit it as two 32-bit integers.
(Surprisingly few languages manage to provide any better alternatives)
Incidentally, another type that inconviniently cannot be portably printf()ed is time_t. This sort of problem is common in networking - try getting a float across a network sometime, without losing accuracy in the process. Of course, a lot of people don't realise this and write code that just doesn't get it right, and then can't understand why it only works on some systems.
-
@DZ-Jay said:
@bobday said:
Of course, there's no documentation at all - I've been reverse-engineering the protocol from the server's log output and the source code.
Have you searched for any documentation?
How about this:
That's not a lot of documentation. It's certainly not enough to code to.
I'm still amazed by the need to change the version number every time a
field is added to the protocol. I agree with bobday: this
is an unfortunate example of the darker side of open source.
(Disclaimer: I've never used MythTV. For me, cron + lirc +
lavrec is sufficient.)
-
@iwpg said:
@bobday said:
The worst part is that the backend seems to randomly segfault on valid input. I can't run it in a debugger, because in addition to being the coordinator of the system it handles recording video data (and a debugger slows it down too much to do that).
Couldn't you just let it dump core, then run the debugger on the dump? Not as flexible as debugging the live process, but better than nothing.
/facepalm
I'd forgotten completely about core dumps. I'll go find out how to enable them, that should help a bit.
@danio said:@bobday said:
Next, ANNounce your intention to get a file:43 ANN FileTransfer Desktop[]:[]/some_file.png
You will get a string in response along these lines:25 OK[]:[]20[]:[]0[]:[]13987
This means that your transfer id is 20 and the file is 13987 bytes long.
You must re-send the ANNounce message to be able to read from the server:21 ANN Monitor Desktop 0
Finally, you can begin reading data (say, 42 bytes worth):46 QUERY_FILETRANSFER 20[]:[]REQUEST_BLOCK[]:[]42
At this point, with everything else in the protocol being performed with strings, you might think that perhaps the data will be encoded into hex or base64 or something. Nope, straight bytes across the wire, with no length prefix (no way of knowing how long the data will be). After the data has been sent, the server will tell you how much was sent - gee, thanks, that would have been nice to know before I had to allocate a buffer to hold it all.
Didn't you just ask it for 42 bytes of a 13987 byte transfer so that should give you enough to work out your buffer size?
Probably the size after transfer is a check so that you can compare how many it sent with how many it should have sent?
Seems pretty cumbersome but no worse than commercial systems I've worked with. At least you have access to the source code, whereas with badly documented commercial products it is just a matter of trial and error till you figure out what the documentation could have made clear in the first place...
What happens if the file is truncated in between sending ANN FileTransfer and getting the data block? There's no way to know that you've hit EOF.
@DZ-Jay said:@bobday said:
Of course, there's no documentation at all - I've been reverse-engineering the protocol from the server's log output and the source code.
Have you searched for any documentation?
How about this:
http://winmyth.sourceforge.net/mythprotocol.html
That is for the very first version of the protocol, although it contains the notice that it might work for version 8. The current stable version is 26, with development at 30. More recent docs are in a wiki at http://www.mythtv.org/wiki/index.php/Myth_Protocol_Command_List, and most of those were added by me.
@asuffield said:@bobday said:
Of course, although it appears to be four values, that's actually only two - 64-bit integers are encoded as two 32-bit integers for some reason, which is bizarre, because they're sent as strings anyway!
Quick test then: how do you convert a 64-bit integer to a string in C?
If you said printf("%lu", x) or something along those lines, and you're using i386, then you're out of luck. long int is only 32 bits on that platform on all the current OSes. And therein lies the problem. GNU C provides long long int, but that isn't portable. C99 provides int64_t, but doesn't provide a printf tag for handling 64-bit values. The fastest, most portable way to transmit a 64-bit integer as a string is to transmit it as two 32-bit integers.
(Surprisingly few languages manage to provide any better alternatives)
Incidentally, another type that inconviniently cannot be portably printf()ed is time_t. This sort of problem is common in networking - try getting a float across a network sometime, without losing accuracy in the process. Of course, a lot of people don't realise this and write code that just doesn't get it right, and then can't understand why it only works on some systems.
It's pretty easy to create a function to serialize or de-serialize a arbitrarily sized integer. It doesn't even have to be in C, both the official library and mine are written in C++.
-
@bobday said:
It's pretty easy to create a function to serialize or de-serialize a arbitrarily sized integer. It doesn't even have to be in C, both the official library and mine are written in C++.
So every client developer is condemned to having to reimplement their standard library just to work with the protocol? Not a very good plan. Most people just send multiple 32-bit integers. Everybody can handle those easily.
-
@asuffield said:
@bobday said:
It's pretty easy to create a function to serialize or de-serialize a arbitrarily sized integer. It doesn't even have to be in C, both the official library and mine are written in C++.
So every client developer is condemned to having to reimplement their standard library just to work with the protocol? Not a very good plan. Most people just send multiple 32-bit integers. Everybody can handle those easily.
If official library had been sane and usable, I wouldn't be re-implementing it.
-
@danio said:
@bobday said:
Next, ANNounce your intention to get a file:43 ANN FileTransfer Desktop[]:[]/some_file.png
You will get a string in response along these lines:25 OK[]:[]20[]:[]0[]:[]13987
This means that your transfer id is 20 and the file is 13987 bytes long.
You must re-send the ANNounce message to be able to read from the server:21 ANN Monitor Desktop 0
Finally, you can begin reading data (say, 42 bytes worth):46 QUERY_FILETRANSFER 20[]:[]REQUEST_BLOCK[]:[]42
At this point, with everything else in the protocol being performed with strings, you might think that perhaps the data will be encoded into hex or base64 or something. Nope, straight bytes across the wire, with no length prefix (no way of knowing how long the data will be). After the data has been sent, the server will tell you how much was sent - gee, thanks, that would have been nice to know before I had to allocate a buffer to hold it all.
Didn't you just ask it for 42 bytes of a 13987 byte transfer so that should give you enough to work out your buffer size?
Probably the size after transfer is a check so that you can compare how many it sent with how many it should have sent?
Seems pretty cumbersome but no worse than commercial systems I've worked with. At least you have access to the source code, whereas with badly documented commercial products it is just a matter of trial and error till you figure out what the documentation could have made clear in the first place...
If you have the source code, you shouldn't need too much documentation. Even if I did have documentation as well as source code, I would still go through the code to verify the documentation and look for anything that was left out. When I was working with COMUnit, I wanted to use the XML parameter file option, however I was unable to find any documentation on the required structure of the XML document that COMUnit expected. COMUnit was opensource, so I simply downloaded the source and in thirty minutes had the structure down pat. Of course, that was simple XML structure, and nothing as complex as this, but still it IS REALLY NICE to have the source to the program that is taking in your output.
Although that still doesn't excuse the klunky interface.
-
@bobday said:
Unfortunately, it has all the downsides of open source with none of the upsides.
None of the upsides? That's ridiculous. It's under the GPL, which forces the the existence of many upsides.
I agree that it looks really WTFy though. In particular the response strings with unlabeled fields alone is deserving of a front page story.
-
@HeroreV said:
@bobday said:
Unfortunately, it has all the downsides of open source with none of the upsides.
None of the upsides? That's ridiculous. It's under the GPL, which forces the the existence of many upsides.
I agree that it looks really WTFy though. In particular the response strings with unlabeled fields alone is deserving of a front page story.I also agree that club fisted piece of work, but if it had none of the upsides of open source software, you wouldn't have the source code. Improve and submit and if your changes are rejected, fork it!
-
@Some Idiot said:
@HeroreV said:
@bobday said:
Unfortunately, it has all the downsides of open source with none of the upsides.
None of the upsides? That's ridiculous. It's under the GPL, which forces the the existence of many upsides.
I agree that it looks really WTFy though. In particular the response strings with unlabeled fields alone is deserving of a front page story.I also agree that club fisted piece of work, but if it had none of the upsides of open source software, you wouldn't have the source code. Improve and submit and if your changes are rejected, fork it!
Funnily enough, I saw someone on the #mythtv channel mention that they came up with a patch to use named fields for the protocol a few years ago, and the idea was totally rejected. I presume it was this thread here they were referring to...
-
@Alexis de Torquemada said:
WTF? Why can't people get a clue before posting blatant nonsense? C99 does not provide int64_t (that was Microsoft Visual C, anyway), it provides "long long".
-
@Alexis de Torquemada said:
@asuffield said:
And therein lies the problem. GNU C provides long long int, but that isn't portable. C99 provides int64_t, but doesn't provide a printf tag for handling 64-bit values.
WTF? Why can't people get a clue before posting blatant nonsense? C99 does not provide int64_t (that was Microsoft Visual C, anyway), it provides "long long".
You are mistaken; C99 does provides int{8,16,32,64}_t (<stdint.h>, §7.18.1.1). Although they are optional in C99, they are required in SUSv3, so you can rely on them being present on any POSIX-like platform.
C99 also provides intmax_t, which has its own printf precision character, j, so any platform that provides 64-bit ints can also print them with printf("%jd", (intmax_t)i).
-
@DES said:
You are mistaken; C99 does provides int{8,16,32,64}_t (<stdint.h>, §7.18.1.1). Although they are optional in C99, they are required in SUSv3, so you can rely on them being present on any POSIX-like platform.
Sorry, I did a quick search for int64_t in my ISO C document and nothing cropped up (it says intN_t, where N is a placeholder). That said, int64_t is not required in all cases even when long long is present.
-
@Alexis de Torquemada said:
This also means that long long is portable to any C99-compliant compiler.
And actually, the ISO C99 standard says quite clearly in subclause 7.19.6.1 "The fprintf function", paragraph 7:
The length modifiers and their meanings are:
...
ll (ell-ell) Specifies that a following d, i, o, u, x, or X conversion specifier applies to a long long int or unsigned long long int argument; or that a following n conversion specifier applies to a pointer to a long long int argument.
Unfortunately, there is no guarantee that long long isn't just another name for the 32-bit integer type, nor is there a guarantee that it will always be big enough to handle a time_t (implementations can and often do offer larger types, and future versions of the C spec may introduce new larger types). It's the same problem you have with size_t, and the reason why you'll find the 'z' length modifier in that section.
I went all over this one a few years back, trying to get a straight answer about how it was expected to work from the various committees, and basically it isn't expected to work. The root problem is that time_t is a POSIX type, while printf is a C function, and the POSIX and C committees don't work together all that closely - C is, after all, used in many non-POSIX environments. There is no provision at all for using printf on POSIX types in a portable fashion - time_t, uid_t, off_t, and all the rest. The C committee is not responsible for handling time_t, and the POSIX committee is not responsible for defining printf, so neither of them is 'allowed' to fix it.
You can write code that works now by casting to long long, but at some point in the future, when C introduces new larger types and some POSIX implementers make their types larger, your code will break.
Currently we have code that's broken because it cast time_t to long, which was correct until about six years ago. The next one we're likely to run into is off_t, when it increases from 64-bit to 128-bit - that should happen in the next ten years, given the current rate of increase in the size of hard drives.
-
@asuffield said:
You can write code that works now by casting to long long, but at some point in the future, when C introduces new larger types and some POSIX implementers make their types larger, your code will break.
Currently we have code that's broken because it cast time_t to long, which was correct until about six years ago. The next one we're likely to run into is off_t, when it increases from 64-bit to 128-bit - that should happen in the next ten years, given the current rate of increase in the size of hard drives.
You can write code that works now and that will still work in the future by casting to intmax_t and using the j length modifier.
