We want Stuff and we want it Compressed.



  • Okay, another open-source project, another homebrew project, so this probably isn't front-page-worthy. Besides, this thing is almost logical; it just gets hideously weird if you actually take your Goggles off.

    The Weird File Format of the Day is OMOD, from Oblivion Mod Manager. Basically, mod package manager for TES4: Oblivion - quite famous, quite well known, and you see plenty of these files around and you can make installing and uninstalling tons of files easy and simple, or so the theory goes. You can even script the installation. I just have a small beef with the format because the OBMM program doesn't run all that well in Linux. =/

    Please note that I'm not really criticising the program per se - it's probably a fine program if you can get it running. I have nothing against the implementors. I'm just criticising the fact that apparently no one has tried to write an alternate implementation of the format and cried, or even looked at the spec to see the most obvious problems that make implementing the format more difficult.

    So, I failed to run OBMM using Mono. I failed to run OBMM using Mono through Wine. I so far haven't gathered up enough courage to install Microsoft's .NET runtime in Wine (and I wonder if it's even acceptable as far as EULA goes). Anyway, to heck with that - with the file format spec in hand, how hard it's to write a program that dumps all of the archive files to the current directory? I mean, it's just another bundle of data with additional script/image/RTF files?

    Let's see... the package is a .zip file. Okay. Except that if you unzip it, you get a bunch of seemingly useless files. The actual data is stored in files that are not actually in any format that conventional archive tools could read.

    The control file is a rather weird thing. The spec just said there are "strings" here, so I had just a few problems getting implementing a parser at 3 in the morning. When the terrible dawn arose, I realised that the strings are actually stored in whatever format .NET stores them, and since I had absolutely no idea how .NET serialises strings, I had to squint at the files long and hard in the hex editor. In the morning, I finally, finally realised what was going on here. Oh, they're not null-terminated strings or even terminated-with-some-random-non-ASCII-byte strings. They're weird hacked-up Pascalesque strings with length bytes prepended. Got it. Sorry. Just my weird misconceptions.

    So now I can actually parse the control file in my own program. The only remaining mystery is the build date field, which unlike the easily parseable human-readable date in the previous version of the spec is apparently a serialised 64-bit integer, which while not entirely a bad idea, is a little bit weird one. At least the spec could have said what the epoch base is and what units are those, but I suppose that's in some .NET reference somewhere. I've just been too lazy to find out right now. Anyway, I can only assume is future-proofing in the unlikely event that we're installing the mods in the next cycle of existence. (Thanks to Swampy, we all know Unix 32-bit signed time_t is far more than enough, right? =)

    All right, then data.crc/plugins.crc. Incidentally, the spec doesn't mention if these values are to be used on compressed or decompressed data. Since the decoded file lengths of the test file at hand are so much more huge than the actual data file size, I'm assuming decoded. So, I need to decode the data first.

    Okay, let's see, the file is in LZMA. The library I have at hand can do LZMA. Very cool. So, I apparently need to...

    ...somehow feed this file to the decoder... I suppose?... I suppose the "stream" has to be somehow cut and fed to the library. It seems this library only uncompresses one file at time. I guess I need to find another library. Hmm...

    ...Well, I think, at least... But what's this about "First 5 bytes should be fed to Decoder.SetDecoderProperties()"? What's that? I can't see that stuff in my library documentation. Whah? I guess I really need another library that can take Mysterious Binary Strings to set the decoding parameters. This library I have at hand is very inadequate.

    Okay, those are just the micro-WTFs. By now, you've probably guessed where the bigger WTF is. Remove the Goggles! Head a little bit farther and look at the Big Picture. Aiigh!

    This is essentially an archive format that is implemented within an existing archive format, .zip, with vague specification on how the files can be extracted from it. The actual files can't be extracted from the package using any well-known tool, nor does the specification present any really big rationale for packaging the data in this particular way. In a reasonably well-designed world, the whole package would have been just a .zip or .7z file. Not here though!



  • Amazing the torture some people will endure just to avoid a Windows installation.  Keep on stickin' it to the man!



  • Wow.... ok, see this is why Microsoft says:

     Don't use binary serialization if you ever might send this data to another operating system or programmming environment.

     Then they give you two alternatives that write the data to text (SOAP or more generic XML), and about 50 good ways to write your own serialization class (which isn't as hard as you'd think, or at least not as hard as displaying blank window in their old Win16 API).

     I'm not sure that I understand your explanation of how they managed to break the ZIP file format, but I would think that they could have found a good library out there that would do the hard work for them.



  • Hmm, if this is completely correct, then it is quite odd that they ended up creating their own archive format and then adding it to a zip.  Still, it's internal to a piece of software so it's sort of like sausage -- you don't really want to know what's going on so long as it works.  It doesn't sound like it was intended to be reverse-engineered, although I find it odd that there was no .NET library available to handle their file format. 



  • @morbiuswilters said:

    then it is quite odd that they ended up creating their own archive format and then adding it to a zip

    Not counting the "own format" business, at least we can be thankful they didn't try to implement their own compression algorithm/utility and just used Zip's facilities, rather like .tar.gz. 



  • @WWWWolf said:

    The control file is a rather weird thing. The spec just said there are "strings" here, so I had just a few problems getting implementing a parser at 3 in the morning. When the terrible dawn arose, I realised that the strings are actually stored in whatever format .NET stores them, and since I had absolutely no idea how .NET serialises strings, I had to squint at the files long and hard in the hex editor. In the morning, I finally, finally realised what was going on here. Oh, they're not null-terminated strings or even terminated-with-some-random-non-ASCII-byte strings. They're weird hacked-up Pascalesque strings with length bytes prepended. Got it. Sorry. Just my weird misconceptions.
     

    Sorry to rain on your mug, but the string and date formats for .net are document.  Likely, the original program uses a BinaryWriter.



  • @flipperanubi said:

    Wow.... ok, see this is why Microsoft says:

     Don't use binary serialization if you ever might send this data to another operating system or programmming environment.

     <hints id="hah_hints"></hints>
    But it [I]wasn't[/I] intended to be sent to another operating system or programming environment, that's the whole point.  It's a mod manager for a Windows game.  How much more platform-specific can you get?


  • @Aaron said:

    Amazing the torture some people will endure just to avoid a Windows installation.  Keep on stickin' it to the man!

    And this is related to my Linux use exactly... how?

    You can't uncompress the file on Windows, either, without using OBMM. The data in the OBMM files does not require OBMM to be installed, either; You can, say, install Oblivion, install OBMM, then some mod, and uninstall OBMM - and the mod still works. So aside of the fact that it was probably easier to implement this way using this particular platform, what justification is there for OBMM file format to be tied to the application?

    @flipperanubi said:

     Then they give you two alternatives that write the data to text (SOAP or more generic XML), and about 50 good ways to write your own serialization class (which isn't as hard as you'd think, or at least not as hard as displaying blank window in their old Win16 API).

    Yep, and I'm not saying this program is unique in this regard.

    For example, don't get me started on what I once saw when I opened OmniOutliner file (not an exported "XML" file, which was sane, but the "internal" plist file, which was just about as far from sane as you can possible get. =)

    @flipperanubi said:

     I'm not sure that I understand your explanation of how they managed to break the ZIP file format, but I would think that they could have found a good library out there that would do the hard work for them.

    What I meant was that they probably decided that .zip and .7z files just don't cut it anymore, came up with a file format that satisfies their requirements, yet it uses the .zip format as the container while not using .zip format's ability to, you know, store files. Instead, the .zip structure is only storing the container information and the actual files in the archive are in a puzzling binary lump...

    @arty said:

    Sorry to rain on your mug, but the string and date formats for .net are document.  Likely, the original program uses a BinaryWriter.

    Did I say they are undocumented? Sorry, I just meant they're not documented in the alleged documentation of this file format. For example, I did figure out the Pascal-string mystery after seeing some Mono and MSDN pages on the topic - otherwise, I'd be still sitting here thinking "okay, we have byte sequence 0xE005 indicating string length, how the hell that can mean 736 in decimal", but MSDN did explain how the integers are packed... Weird, but not unheard of.



  • Oh, I assumed since he was extracting the files on Linux the game had a Linux port. Guess that's a WTF on me. :)

     



  • There's no WTF here -  I believe OBMM is designed that way. 

    You're not supposed to be able to open an OMOD file with anything other than OBMM. If you start using zip files, or any common format, one of two things will happen:

    1) Naive users will think that every zip file can be used with OBMM. Hilarity will ensue and modders (and the OBMM creators) will be blasted with emails from idiots because their non-OMOD zip files don't work with OBMM. As a modder, I wouldn't want that to happen.

    2) Naive users will think that they can edit the contents of the OMOD zip file, and things will still work. These are the same type of people that delete things from the root folder of their C:/ drive to save space. Even worse, these people will try to distribute their non-working versions of the zip file, as the real thing. Again, hilarity will ensue and a lot of drama will commence.

    I'm not even going to ask why you're trying to run OBMM under Linux, because to play Oblivion, you've got to have a slam-dunk Windows PC around somewhere, so why don't you just run OBMM perfectly capably on that? 



  •  I guess TRWTF is I spent my nights playing Oblivion instead of going over it with a hex editor. :)



  • @Becky said:

    You're not supposed to be able to open an OMOD file with anything other than OBMM. If you start using zip files, or any common format, one of two things will happen:

    ...or not. Don't forget that tons of Oblivion mods are distributed as ordinary .zip or .7z files. Also, tons of other games use .zip files for mod packaging (id .pkX files spring in mind immediately). And do you see a lot of people complaining "I can't run this zip file with Java?" ...'cause that's what .jar files are. =)

    @Becky said:

    I'm not even going to ask why you're trying to run OBMM under Linux, because to play Oblivion, you've got to have a slam-dunk Windows PC around somewhere, so why don't you just run OBMM perfectly capably on that? 

    The game does run under Linux. Not as fast as on Windows (not that I care that much, my computer is pretty underpowered to run the game as such anyway) and there's several annoying glitches, and of course installation needs some special care. But the bottom line is, all you need is stock Wine and one DLL that can be found on the install DVD. =)



  • @WWWWolf said:

    .... SNIP ...

    Anyway, I can only assume is future-proofing in the unlikely event that we're installing the mods in the next cycle of existence. (Thanks to Swampy, we all know Unix 32-bit signed time_t is far more than enough, right? =)

    ... SNIP ...

     

    Hilarious. But is there a magic stone somewhere ? 



  • I know that tons of mods are released as 7z or zip format. My comment was, that if both OBMM and normal mods were packaged in the same format, a lot of novice users (ands for a game like Oblivion... that's a lot of users) would be getting seriously mixed up, and fair enough too. Instead of something like Better Bodies.omod and Better Bodies.zip, you'd have things like Better Bodies (OBMM).zip and Better Bodies (non OBMM).zip.... and trusting people to keep things to any sort of naming convention is dangerous indeed.

    And I didn't actually know that Oblivion will run under WINE, cheers for that. I'll stick to playing it under XP though :)



  • There's no reason not to have it just be a zip file by a different name that extracts to the files, rather than "zip containing weird archive format containing real files", though, which was the point.

    After all, there are many "file formats" that are really just zip (though the contents have a certain structure) - jar, xpi, wsz, wmz, pk3, pk4, kmz, nth, docx, xlsx, sometimes exe (those self-extractors can always be opened in any zip utility), gif/jpg (steganography, you know), open document (what extension?), and many more.



  • @Random832 said:

    open document (what extension?)
     

    .od[tspgf] 


Log in to reply