Typo in EVE Online patch results in Windows installation corruption



  • Linkage

    WTF 1: Naming a game data file "boot.ini"

    WTF 2: Accidentally adding a backslash, resulting in the patch deleting "\boot.ini", aka "C:\boot.ini".

    WTF 3: This getting through QA and ending up in a patch run by up to two hundred thousand people.

    WTF 4: The fact that Windows allows applications to overwrite boot.ini. 



  • Criminey, I came to submit this to WTF and right after I click Submit I see this in the sidebar. By Shikari of eve-o fame, no less :(

    That out of the way, I'm filled with hate for CCP's development and QA teams, but I kind of feel bad for CCP as a whole. This patch was going to be huge, kind of a crowning glory, and now it's completely ruined for thousands of people by this issue--some of whom will inevitably give money to Geek Squad or something to get their computer fixed. 



  • For once something that did got better in Vista ;)



  • I am not too sure Vista will prevent this either.
    I mean, the installer does need the Admin privilege token if it plans on writing to %programfiles%...



  • @henke37 said:

    I am not too sure Vista will prevent this either. I mean, the installer does need the Admin privilege token if it plans on writing to %programfiles%...

    But they got rid of the boot.ini, it's now managed with bcdedit, it's stored in the c:\boot\ folder.



  • Awful. Truly, truly awful. If CCP are being this sloppy with their game's code, are they making similarly stupid mistakes with the code that handles the billing?

    Also, this makes me think that an operating system should be designed such that most software can be installed by an account with less than the administrator/root account. In fact I'm sure one could implement this in Linux if one desired. At the moment Linux would be just as vulnerable to this kind of screwup by a third-party developer as Windows has shown itself to be.

    And it's just another reason to always use absolute paths.



  • @m0ffx said:

    Also, this makes me think that an operating system should be designed such that most software can be installed by an account with less than the administrator/root account. In fact I'm sure one could implement this in Linux if one desired. At the moment Linux would be just as vulnerable to this kind of screwup by a third-party developer as Windows has shown itself to be.

    I'm not familiar with any mainstream package management systems that let you do this, but for userland software that you have to install by hand anyways (unrealircd, ventrilo server, soldat dedicated server, etc), I'm very careful not to let them run any code under anything except the user account created specifically for them. For software that wants you to run `make install` as root (to write to /etc/, /usr/lib/, et all), you can usually work around that by ./configure'ing it differently and modifying your ld settings (LD_LIBRARY_PATH).

     It's absolutely true that this isn't something that your stereotypical ubuntu user would do, though.
     



  • It's a horrible mistake, but it didn't affect 200,000 people, only the people who downloaded and installed it between 10pm (or whenever the game came back up) and about 8am (possibly a few hours earlier). It was due to come back up at 2am, so most people would have waited until after it was fixed. It also only applied to the classic->premium upgrade patch. The premium full installer and the old build->trinity classic patch were both fine.

    If you have Windows XP SP2 installed, then it can survive boot.ini being missing (also applies to XP x64, I know that first-hand).

    I think probably only a dozen or so people actually had their pc rendered unbootable by this and couldn't fix it themself. Still, that's a dozen too many. 



  • @Thief^ said:

    It's a horrible mistake, but it didn't affect 200,000 people, only the people who downloaded and installed it between 10pm (or whenever the game came back up) and about 8am (possibly a few hours earlier). It was due to come back up at 2am, so most people would have waited until after it was fixed. It also only applied to the classic->premium upgrade patch. The premium full installer and the old build->trinity classic patch were both fine.

    That's why I said "up to".  I'd guess the actual numbers are around a few tens of thousands.



  • @m0ffx said:

    Awful. Truly, truly awful. If CCP are being this sloppy with their game's code, are they making similarly stupid mistakes with the code that handles the billing?

    Also, this makes me think that an operating system should be designed such that most software can be installed by an account with less than the administrator/root account. In fact I'm sure one could implement this in Linux if one desired. At the moment Linux would be just as vulnerable to this kind of screwup by a third-party developer as Windows has shown itself to be.

    And it's just another reason to always use absolute paths.

    On Linux you can install programs as user, but you must have write permission for the installation dirs (ie. you can install it in your $HOME). I have KDE4 (svn) installed in my $HOME, and the only thing that require root is Korundum (ruby bindings) because it needs to install files in /usr/local/lib/ specifically.

    Even though it's possible I don't know of any package manager that's able to do it (not that I looked for this feature anyway).



  • @Dark Shikari said:

    WTF 3: This getting through QA and ending up in a patch run by up to two hundred thousand people.

    WTF 4: The fact that Windows allows applications to overwrite boot.ini.

    WTF 3:  Either the patch didn't go through QA or there are some QA people polishing their resumes right now.

    WTF 4:  Linux doesn't stop you from rm -rf *  in / when you're root.  Why should Windows be different?  Also, what if a legitimate Windows patch needs to update the file? 



  • @krupa said:

    WTF 4:  Linux doesn't stop you from rm -rf *  in / when you're root.  Why should Windows be different?  Also, what if a legitimate Windows patch needs to update the file? 

     You can argue that you should not be allowed to do so, unless you explicit allow the process to do so, that would prevent most accidental modification of the important files.
     



  • Sure, and User Account Control essentially does that on Vista (yeah, yeah, UAC sucks, Vista sucks, etc.).  All I was trying to say is that it's not really a WTF in my opinion.  There are legitimate reasons (maybe) to write to boot.ini and it's not just Windows that lets you kill your computer by corrupting/overwriting critical files.



  • Thanks for the heads-up. I hadn't rebooted yet, so didn't realize what they'd done to me.

    Good thing I was able to add a new boot.ini before I rebooted, people who built this system
    never gave me a windows cd to fix a broken boot with.



  • Something tells me they didn't test this patch very well.



  • I always complain about installers that try to force me to reboot before I can do anything else, but one of those may have come in handy here.  I can easily see the cause of this being a tester who leaves his machine running 24/7.



  • I got hit by this on 2 different computers.  I'm glad I caught it before I rebooted my laptop, as I don't think I could have gotten it back.

     

    On the one hand, my angry side wants someone to be fired, then shot, then dipped in acid for this, but my rational side realizes that the programmer who made this mistake is the least likely to ever make another mistake of this kind on the team.



    For all its flaws, at least EVEMon never nuked anyone's machine :) 



  • @Anders Chydenius said:

    On the one hand, my angry side wants someone to be fired, then shot, then dipped in acid for this, but my rational side realizes that the programmer who made this mistake is the least likely to ever make another mistake of this kind on the team.

    Everybody makes typos, that's the least of the WTFs here.  The person who most deserves to be dunked into a pool of piranha-vipers over this is the QA guy who failed to reboot the machine after the installation, and/or the QA lead who neglected to add a reboot to the testing procedure.
     



  • @seaturnip said:

    @Anders Chydenius said:

    On the one hand, my angry side wants someone to be fired, then shot, then dipped in acid for this, but my rational side realizes that the programmer who made this mistake is the least likely to ever make another mistake of this kind on the team.

    Everybody makes typos, that's the least of the WTFs here.  The person who most deserves to be dunked into a pool of piranha-vipers over this is the QA guy who failed to reboot the machine after the installation, and/or the QA lead who neglected to add a reboot to the testing procedure.

    I don't work on desktop apps -- is that normally a step one would take in a patch testing process?  



  • @shadowman said:

    @seaturnip said:
    @Anders Chydenius said:

    On the one hand, my angry side wants someone to be fired, then shot, then dipped in acid for this, but my rational side realizes that the programmer who made this mistake is the least likely to ever make another mistake of this kind on the team.

    Everybody makes typos, that's the least of the WTFs here.  The person who most deserves to be dunked into a pool of piranha-vipers over this is the QA guy who failed to reboot the machine after the installation, and/or the QA lead who neglected to add a reboot to the testing procedure.

    I don't work on desktop apps -- is that normally a step one would take in a patch testing process?  


    I wouldn't know. The procedure here when testing installers is to run a filesystem diff after installing the software, and compare the results against a list of what files should have changed.



  • @Carnildo said:

    @shadowman said:
    @seaturnip said:

    Everybody makes typos, that's the least of the WTFs here.  The person who most deserves to be dunked into a pool of piranha-vipers over this is the QA guy who failed to reboot the machine after the installation, and/or the QA lead who neglected to add a reboot to the testing procedure.

    I don't work on desktop apps -- is that normally a step one would take in a patch testing process?  


    I wouldn't know. The procedure here when testing installers is to run a filesystem diff after installing the software, and compare the results against a list of what files should have changed.

    I've worked on desktop applications, but installers are outsourced to a specialized team, so I'm not sure of our company's exact process with it.

    Though, yeah, a filesystem diff would seem to catch pretty much any problem that could interfere with a reboot, and then some.  Still, it seems like such a obvious step to attempt a reboot after an install or uninstall that it boggles my mind that it wouldn't be done as a matter of course for non-urgent patches.  Just in case the application, say, has some dependency on a DLL loaded by the installer or something and fails to start after a reboot.



  • @m0ffx said:

    Awful. Truly, truly awful. If CCP are being this sloppy with their game's code, are they making similarly stupid mistakes with the code that handles the billing?

    As a general rule, all software related to games is appallingly badly written. It's to do with their attitude towards scheduling and deadlines, and the fact that nowadays it's a "release once, release one or two patches, and then abandon all maintenance forever" system.



  • @Dark Shikari said:

    Linkage

    WTF 1: Naming a game data file "boot.ini"

    WTF 2: Accidentally adding a backslash, resulting in the patch deleting "\boot.ini", aka "C:\boot.ini".

    WTF 3: This getting through QA and ending up in a patch run by up to two hundred thousand people.

    WTF 4: The fact that Windows allows applications to overwrite boot.ini. 

     

    Owned. 



  • @asuffield said:

    @m0ffx said:

    Awful. Truly, truly awful. If CCP are being this sloppy with their game's code, are they making similarly stupid mistakes with the code that handles the billing?

    As a general rule, all software related to games is appallingly badly written. It's to do with their attitude towards scheduling and deadlines, and the fact that nowadays it's a "release once, release one or two patches, and then abandon all maintenance forever" system.

    This is why Blizzard is held in such high regard. They release once, release a hundred patches and never stop fixing bugs and more bugs! Whether this is a good or bad thing is still up in the air.

    After the recent horrible launch of Flagship's Hellgate:London (think several quest bugs that rendered you unable to proceed if you crashed at the wrong moment, combined with a guaranteed crash due to memory leak after X time... and for good measure throw in a few bugs that eat your equipped weapons and one funny bug that randomly teleports you out of a safe town into the middle of a battlezone when you're AFK) after a 3 month alpha and beta, I'm convinced that game developers don't have bad QA... they just don't bother actually fixing the bugs.



  • @Brother Laz said:

    This is why Blizzard is held in such high regard. They release once, release a hundred patches and never stop fixing bugs and more bugs! Whether this is a good or bad thing is still up in the air.


    No, they're held in high regard because their multiplayer games are addictive and have some soul.

    As for their when-it's-done development process, it's fine for their main market but note how they've failed in the console market.  Starcraft Ghost took so long to develop it missed a console generation, needed to be completely rewritten, therefore missed the next console generation, and finally was canned.  Goes to show that the game industry's dominant attitude towards scheduling has some sense to it.



  • StarCraft Ghost wasn't being developed by Blizzard.  They had a third party working on it (first Nihilistic, then Swingin' Ape).  Then, Blizzard acquired Swingin' Ape, but mid-development changes in management caused development hell that ultimately caused the game to be indefinitely postponed.

     

    That said, as a game developer, Blizzard is held in high regard because they eschew the EA-like game development sentiment of "crunch time means 80hr weeks for 3 months until we get the game out."  Also, perpetually updating games like eve, world of warcraft, everquest, etc. are reputed to have better codebases than console games or non-updating games because, as updating games they are going back to the source code, whereas other games are "ship it and move on to the next". 



  • @asuffield said:

    @m0ffx said:

    Awful. Truly, truly awful. If CCP are being this sloppy with their game's code, are they making similarly stupid mistakes with the code that handles the billing?

    As a general rule, all software related to games is appallingly badly written. It's to do with their attitude towards scheduling and deadlines, and the fact that nowadays it's a "release once, release one or two patches, and then abandon all maintenance forever" system.

    This statement does not apply to MMOGs for obvious reasons.



  • @ebs2002 said:

    Also, perpetually updating games like eve, world of warcraft, everquest, etc. are reputed to have better codebases than console games or non-updating games because, as updating games they are going back to the source code, whereas other games are "ship it and move on to the next".


    I'm not sure about that.  I'm sure they [i]tried[/i] to make it more maintainable than average, but I've seen a lot of evidence that MMORPG teams are even less organized than usual.



  • @seaturnip said:

    @Anders Chydenius said:

    On the one hand, my angry side wants someone to be fired, then shot, then dipped in acid for this, but my rational side realizes that the programmer who made this mistake is the least likely to ever make another mistake of this kind on the team.

    Everybody makes typos, that's the least of the WTFs here.  The person who most deserves to be dunked into a pool of piranha-vipers over this is the QA guy who failed to reboot the machine after the installation, and/or the QA lead who neglected to add a reboot to the testing procedure.
     

     

    If the patch corrupted your Microsoft Office installation, would you blame the QA tester for neglecting to add running Word after patching to the testing procedure? 



  • @shadowman said:

    I don't work on desktop apps -- is that normally a step one would take in a patch testing process?  

    If you designed your installation to ask for a reboot (which is totally your option), I'd expect you to have a reason for needing a reboot, and therefore the common sense to test a reboot. 



  • @tster said:

    If the patch corrupted your Microsoft Office installation, would you blame the QA tester for neglecting to add running Word after patching to the testing procedure? 

    If the patch corrupted your Office installation, it wouldn't render your system broken and non-booting. 



  • @ebs2002 said:

    Also, perpetually updating games like eve, world of warcraft, everquest, etc. are reputed to have better codebases than console games or non-updating games because, as updating games they are going back to the source code, whereas other games are "ship it and move on to the next". 

    That's variable - WoW might, but the other two not so much, and there are a lot of fire-and-forget MMOGs around from people just trying to cash in. Also, MMOGs have a notable reputation for being in a "permanent beta" state, and for using a developmestucation system.


Log in to reply