In 2076 we will be able to put 3GB files on 2GB flash drives



  • When my mother plugged in her 2 GB USB flash drive today, she discovered that some of her files had traveled back in time to warn her of the impending failure of mathematics:

    Gobbledygook



  • I saw a noticeable uptick in this kind of failure when we started deploying Windows 7 on new school workstations instead of Windows XP. Microsoft should never have hidden the "safely remove hardware" system tray icon by default; most people accustomed to using that have no reason to notice the "eject" option in the right-click menu for removable devices, and assume that the lack of a visible "safely remove hardware" button means that this is something you no longer need to do with Windows 7.

    Adding a logon-script registry tweak to make all system tray icons visible by default has reduced the problem back to XP levels, where it once again affects only the kind of person who will not do the "safely remove" dance because "it's always worked fine without that" until it spectacularly doesn't. That's the same kind of person who, though gushingly, pathetically grateful after I spend several hours recovering vital files saved only on their never-backed-up traveling USB stick, will not make that gratitude meaningful by taking my repeated "safely remove hardware" training and backup advice to heart.

    This is a social problem, not a technical one; no amount of careful OS design will ever stop people like that finding ways to hurt themselves with technology. Fortunately they only comprise about 10% of the user base.



  • In this case, I think the particular user (my mother) was not at fault; it was just a sporadic failure. And this drive was apparently actually a "backup drive", with the canonical data being on her laptop's hard disk. But that doesn't mean that what you're saying doesn't account for a significant percentage of problems in the general user base.

    I just had the Micro SD card in my phone give out a couple weeks ago, albeit not so spectacularly. Based on previous behavior (random files getting corrupted) and failure mode (detecting the presence of an SD card but saying it's unformatted), I suspect it's been decaying slowly for a year or so and finally flipped one too many bits in some table somewhere—i.e. the data might still be there, I just need to find a program that can autopsy the file system.



  • I've had fairly good results with ZAR.



  • @flabdablet said:

    Microsoft should never have hidden the "safely remove hardware" system tray icon by default; most people accustomed to using that have no reason to notice the "eject" option in the right-click menu for removable devices, and assume that the lack of a visible "safely remove hardware" button means that this is something you no longer need to do with Windows 7.

    You mean there are other people in the world who know you shouldn't just pull things like USB sticks out of computers when you're done with them?



  • @flabdablet said:

    I saw a noticeable uptick in this kind of failure when we started deploying Windows 7 on new school workstations instead of Windows XP. Microsoft should never have hidden the "safely remove hardware" system tray icon by default; most people accustomed to using that have no reason to notice the "eject" option in the right-click menu for removable devices, and assume that the lack of a visible "safely remove hardware" button means that this is something you no longer need to do with Windows 7.

    Adding a logon-script registry tweak to make all system tray icons visible by default has reduced the problem back to XP levels, where it once again affects only the kind of person who will not do the "safely remove" dance because "it's always worked fine without that" until it spectacularly doesn't. That's the same kind of person who, though gushingly, pathetically grateful after I spend several hours recovering vital files saved only on their never-backed-up traveling USB stick, will not make that gratitude meaningful by taking my repeated "safely remove hardware" training and backup advice to heart.

    This is a social problem, not a technical one; no amount of careful OS design will ever stop people like that finding ways to hurt themselves with technology. Fortunately they only comprise about 10% of the user base.

     

    Windows should just do a sync on each file copy, so you CAN pull the flash drive out safely.

    But that would only train users to never "safely remove". Maybe a better idea is to intentionally corrupt the FAT on insert and uncorrupt on "safely remove" -- that ought to train users.

     



  • @flabdablet said:

    Microsoft should never have hidden the "safely remove hardware" system tray icon by default; most people accustomed to using that have no reason to notice the "eject" option in the right-click menu for removable devices, and assume that the lack of a visible "safely remove hardware" button means that this is something you no longer need to do with Windows 7.

    I thought they made it better in Windows 7, including removing write caching to these sort of media, so everything should be complete when the progress bar disappears. That said, I have corrupted far too many external drives by not "removing safely" on Windows, Mac and Linux (whether purpose or accident), so I am more careful these days!



  •  I have never once corrupted a USB stick by prematurely removing.

    Maybe this is because the fault can only occur when you explicitly yank the stick at the very instant the copy dialog vanishes, and I never do that?



  • @Mo6eB said:

    Maybe a better idea is to intentionally corrupt the FAT on insert and uncorrupt on "safely remove" -- that ought to train users.

    As a compromise, how about setting a bit that tells the OS that the drive was removed improperly, and when it is detected on insertion having a dialog pop up saying, "This drive was removed without ejecting and may be corrupt." Then it should run a very slow disk check (progress bar moving for ~5 mins, even if it only take 30 secs to complete), with a modal dialog that includes instructions for proper removal. That might help slightly increase the number of users who do it properly.



  • @flabdablet said:

    This is a social problem, not a technical one; no amount of careful OS design will ever stop people like that finding ways to hurt themselves with technology.
     

    Not with technology  overall... But the problem of drivers corrupting themselves if not properly removed is cause by the lack of journaling on the filesystem. There is no excuse to corrupt anything besides the file that you were accessing at the time of removal.

    (And, yes, like several other widespread usability problems, that's Microsoft's fault.)



  • @Zemm said:

    I thought they made it better in Windows 7, including removing write caching to these sort of media, so everything should be complete when the progress bar disappears. That said, I have corrupted far too many external drives by not "removing safely" on Windows, Mac and Linux (whether purpose or accident), so I am more careful these days!

    AFAIK Windows has always used a default write-through caching policy on removable devices. Even so, it doesn't react well to having a device removed when there are files open on it, which is the main thing Safely Remove Hardware will catch.



  • @Mo6eB said:

    Windows should just do a sync on each file copy, so you CAN pull the flash drive out safely.

    They already do. Even in XP. (I think 2000 didn't.) Or was this some high-brow humor I didn't get?

    The "problem" isn't that Windows isn't doing its part, it's that people pull the drive while the activity light is still flashing.

    Putting the warning back would be useless because everybody ignores it. You can't change the people, only the technology.



  • @Mcoder said:

    (And, yes, like several other widespread usability problems, that's Microsoft's fault.)

    Praytell explain to us how.

    Are you asserting that NTFS is not a journaling filesystem? Or are you exposing some other undoubtedly hilarious misunderstanding you have about the world?



  • @blakeyrat said:

    Are you asserting that NTFS is not a journaling filesystem?

    I see a lot more USB flash drives formatted with FAT16 or FAT32 than NTFS, just because that's generally the way they come from the jellybean factory and most people don't know how to or don't see a need to format drives themselves; the FAT filesystems don't do journaling, and do tend to fare worse from unsafe removal as a result.

    Lots of jellybean drives have no activity light either.



  • @flabdablet said:

    I see a lot more USB flash drives formatted with FAT16 or FAT32 than NTFS, just because that's generally the way they come from the jellybean factory and most people don't know how to or don't see a need to format drives themselves; the FAT filesystems don't do journaling, and do tend to fare worse from unsafe removal as a result.

    Lots of jellybean drives have no activity light either.

    And this is Microsoft's fault because.........................................?



  • @flabdablet said:

    Lots of jellybean drives have no activity light either.

    I bought a 8GB sandisk flashdrive last week. I was surprised and kinda annoyed to find out it didn't have an activity light, unlike all my previous kingston flash drives I've had on the last 7-8 years. Without it, how the hell am I supposed to know when to remove it from the computer? A lot of times, on many different computers, the message "you can safely remove bla bla blaaah" shows up WHILE THE LIGHTS ARE STILL FLASHING.

    It also helps to notice possible errors, such as when my previous flash drive was slowly turning into a useless piece of plastic, the lights would blink a lot, very very quickly.


  • :belt_onion:

    @Mcoder said:

    There is no excuse to corrupt anything besides the file that you were accessing at the time of removal.
    And what if you would interrupt write process while it was making changes to the file allocation table? Would it then corrupt other things than just the file you were writing to?

    @atipico said:

    the message "you can safely remove bla bla blaaah" shows up WHILE THE LIGHTS ARE STILL FLASHING
    I'm not an expert but I always thought you can safely interrupt 'read' activities on disk. And even during writing when there is nothing left in the cache to be committed it would also be safe to remove.

     



  • @Mo6eB said:

    Windows should just do a sync on each file copy, so you CAN pull the flash drive out safely.
     

    I thought this was a configurable setting: optimise for quick removal, or optimise for fast throughput (cache).

    I had mine on the former so I could yank it once the activity light on the pen stopped flashing (yeah, I only used Kingstons) and haven't experienced corruption.

    I've once encountered an issue when a large filecopy was aborted mid-action and Windows taking ages to cancel the operation, then leaving what appeared to be a large file that couldn't be deleted on the pen. A quick scan fixed that issue, but not sure how it managed to create a ghost copy.



  • @bjolling said:

    I'm not an expert but I always thought you can safely interrupt 'read' activities on disk.
     

    Not on journalled filesystems - sometimes changed changed data needs to be committed (journal flushed) else you'll get a dirty read.



  • @blakeyrat said:

    @Mcoder said:
    (And, yes, like several other widespread usability problems, that's Microsoft's fault.)

    Praytell explain to us how.

    Are you asserting that NTFS is not a journaling filesystem? Or are you exposing some other undoubtedly hilarious misunderstanding you have about the world?

     

    Yeah, let's just forget that MS recomends that removable media be formated as FAT32.

    Anyway, TRWTF is NTFS. It has a journal, but somehow can get corupted if a writting is interrupted. WTF does it write on the journal? Recent news?

     



  • @bjolling said:

    @Mcoder said:
    There is no excuse to corrupt anything besides the file that you were accessing at the time of removal.
    And what if you would interrupt write process while it was making changes to the file allocation table? Would it then corrupt other things than just the file you were writing to?
     

    First, most filesystems don't use a FAT. It is too fragile a thing.

    Now, about your question. A proper journaled filesystem is never in a corrupted state. You can interrupt it any time, the wrost that will happen is that you'll lose the files you are writting to (some protect even against that) and you may have to wait a few seconds while the journal is rolled back.



  • @Mcoder said:

    @bjolling said:
    @Mcoder said:
    There is no excuse to corrupt anything besides the file that you were accessing at the time of removal.
    And what if you would interrupt write process while it was making changes to the file allocation table? Would it then corrupt other things than just the file you were writing to?
     

    First, most filesystems don't use a FAT. It is too fragile a thing.

    Except for the fact that 99.9% of all flash drives (which is what we are discussing here) come pre-formatted from the factory as FAT32.  This is done for lowest-common-denominator compatibility purposes, since just about any OS or device from the last 15 years can read/write FAT32.  Although you can format a flash drive with NTFS (and much to my surprise it was readable when I plugged it into my TV) most people don't know / don't care about that.  They just want to copy their porn important files.

     



  • @blakeyrat said:

    @flabdablet said:

    I see a lot more USB flash drives formatted with FAT16 or FAT32 than NTFS, just because that's generally the way they come from the jellybean factory and most people don't know how to or don't see a need to format drives themselves; the FAT filesystems don't do journaling, and do tend to fare worse from unsafe removal as a result.

    Lots of jellybean drives have no activity light either.

    And this is Microsoft's fault because.........................................?

    Did I say it was?



  • @flabdablet said:

    @blakeyrat said:
    @flabdablet said:

    I see a lot more USB flash drives formatted with FAT16 or FAT32 than NTFS, just because that's generally the way they come from the jellybean factory and most people don't know how to or don't see a need to format drives themselves; the FAT filesystems don't do journaling, and do tend to fare worse from unsafe removal as a result.

    Lots of jellybean drives have no activity light either.

    And this is Microsoft's fault because.........................................?

    Did I say it was?

    No, somebody else did: @Mcoder said:
    like several other widespread usability problems, that's Microsoft's fault
    I think Blakey got confused.

     



  • @El_Heffe said:

    I think Blakey got confused.

    So we're up to the part where I get to spew an apoplectic rant on his lack of reading comprehension?

    Cool.



  • @Mo6eB said:

    Windows should just do a sync on each file copy, so you CAN pull the flash drive out safely.
    The reason most operating systems don't do that by default is to prevent excessive writes to a type of memory with a finite number of writes available. Sure, this is less of aproblem with newer devices, but cheaper/older USB sticks still suffer from this. Writing the file back unecessarily too often (like a Word doc autosaving, or just a contientious user hitting the save button often) would use up those available writes.



  • @ASheridan said:

    The reason most operating systems don't do that by default is to prevent excessive writes to a type of memory with a finite number of writes available. Sure, this is less of aproblem with newer devices, but cheaper/older USB sticks still suffer from this.
    Except that every Windows since XP does that by default. If it's removable, the default policy is to not do any write caching.



  • Its not just a journalling filesystem or its absence which causes problems, but also not using FAT formats...

    ...

    Way back we spent time building embedded Linux systems and we used to get an incredibly short lifespan in some cases using USB flash keys or SD cards.

    Sometimes only days before they became toast. (Some would live longer though)

    Eventually we realised that the failures was when they were not FAT formatted but instead some other  filesystem format. Quite often reformatted as FAT they would start working again, or would come back with only half the quoted capacity remaining available.

    So we started looking at this problem and realised that there are patents from the likes of SanDisk on using the fact that every sector on a flash drive has some additional space for error correction and also some spare 'free' bytes. The 'free' bytes are used in some of the patented schemes to help manage write-levelling for FAT file systems, to stop wearing out some sectors faster than others.

    If you dont use FAT the controller in the card or USB key gets confused.

    Our problems went away when we simply made a big (FAT) file in the FAT file system and then mounted that file as a 'disk drive' and formatted the file with whatever Linux filesystem we liked.

    If you look at Linux based system recovery USB keys this is what they do.

    So formatting a USB key as NTFS could lead to complete and rapid failure even if you obey the blinking light and correctly perform the safely eject hardware dance.

     




  • @flabdablet said:

    @El_Heffe said:
    I think Blakey got confused.

    So we're up to the part where I get to spew an apoplectic rant on his lack of reading comprehension?

    Cool.

     

    Bah. That's redundant, not cool.

     



  • @mikedjames said:

    So we started looking at this problem and realised that there are patents from the likes of SanDisk on using the fact that every sector on a flash drive has some additional space for error correction and also some spare 'free' bytes. The 'free' bytes are used in some of the patented schemes to help manage write-levelling for FAT file systems, to stop wearing out some sectors faster than others.

    If you dont use FAT the controller in the card or USB key gets confused.

     

    Recent Linux kernels have fixed this issue. There is a compatibility layer under tha filesystem that deals with this. Unfortunately, I don't know much about it, except that it's there (and there was an interesting flamewar by the time people was discussing if it was needed or not).

    But yes, Windows won't do that. Formating it as NTFS will probably kill the memory.

     



  • Last time I used OSX (Snow Leopard), it couldn't write data to NTFS, it was read-only. I don't know if Apple's shipping a working NTFS driver with their systems now.

    And, IIRC, if Windows doesn't do it's thing to mark a drive as "safely removed", it nags you about checkdisking. (Tho most people don't read dialog boxes anyways. I still say we need some sort of computing license...)



  •  @barfoo said:

    @Mo6eB said:
    Maybe a better idea is to intentionally corrupt the FAT on insert and uncorrupt on "safely remove" -- that ought to train users.
    As a compromise, how about setting a bit that tells the OS that the drive was removed improperly, and when it is detected on insertion having a dialog pop up saying, "This drive was removed without ejecting and may be corrupt." Then it should run a very slow disk check (progress bar moving for ~5 mins, even if it only take 30 secs to complete), with a modal dialog that includes instructions for proper removal. That might help slightly increase the number of users who do it properly.

    So basically, a Windows version of Mr. Resetti.

    I approve.



  • @barfoo said:

    @Mo6eB said:
    Maybe a better idea is to intentionally corrupt the FAT on insert and uncorrupt on "safely remove" -- that ought to train users.

    As a compromise, how about setting a bit that tells the OS that the drive was removed improperly, and when it is detected on insertion having a dialog pop up saying, "This drive was removed without ejecting and may be corrupt." Then it should run a very slow disk check (progress bar moving for ~5 mins, even if it only take 30 secs to complete), with a modal dialog that includes instructions for proper removal. That might help slightly increase the number of users who do it properly.

    Better idea: Set a bit on the drive at the beginning of all IO operations, and clear it on completion. Then, if a new drive is detected with that bit set, it must have been removed at an inappropriate time. At this point, Windows should lock the computer until a 500 word essay is submitted to Microsoft explaining that they know what they did was wrong and what they will to do ensure it doesn't happen again.

    That will definitely improve the user experience.



  • "This is a social problem, not a technical one;"

    No, this is an HR problem. If someone loses something because they won't follow instructions, they should be treated as having done so deliberately. If IT has to spend several hours recovering stuff because they won't follow instructions, their department should be billed for the time, and it should come out of their supervisors' bonus pool.

    But few places will ever do that, especially schools with teacher's unions.



  • @flabdablet said:

    I saw a noticeable uptick in this kind of failure when we started deploying Windows 7 on new school workstations instead of Windows XP. Microsoft should never have hidden the "safely remove hardware" system tray icon by default; most people accustomed to using that have no reason to notice the "eject" option in the right-click menu for removable devices, and assume that the lack of a visible "safely remove hardware" button means that this is something you no longer need to do with Windows 7.

    Adding a logon-script registry tweak to make all system tray icons visible by default has reduced the problem back to XP levels, where it once again affects only the kind of person who will not do the "safely remove" dance because "it's always worked fine without that" until it spectacularly doesn't. That's the same kind of person who, though gushingly, pathetically grateful after I spend several hours recovering vital files saved only on their never-backed-up traveling USB stick, will not make that gratitude meaningful by taking my repeated "safely remove hardware" training and backup advice to heart.

    This is a social problem, not a technical one; no amount of careful OS design will ever stop people like that finding ways to hurt themselves with technology. Fortunately they only comprise about 10% of the user base.

    Solution: pornographic audio at maximum volume for the unsafe removal notification, set via Group Policy.



  • Better option is to disable write buffer for flash stick drive. In this case You can eject it instanly after You complete copying data.
    AFAIK this is default in Win7.


  • Considered Harmful

    @db2 said:

    Solution: pornographic audio at maximum volume for the unsafe removal notification, set via Group Policy.

    Don't mind me, I'll just be at my desk with my headphones on, plugging and unplugging my USB drive repeatedly.



  • @spamcourt said:

    Better option is to disable write buffer for flash stick drive. In this case You can eject it instanly after You complete copying data.
    AFAIK this is default in Win7.

    I could think you've posted here without reading the rest of the thread, but I prefer to think you have some kind of Momento-esque memory condition that prevents you from remembering something that's already been posted in this thread THREE. FUCKING. TIMES.



  • @barfoo said:

    how about setting a bit that tells the OS that the drive was removed improperly,
    They already have this.@barfoo said:
    and when it is detected on insertion having a dialog pop up saying, "This drive was removed without ejecting and may be corrupt."
    They did this in Windows 2000 but removed it in Windows XP due to a large number of complaints, most of which were "This box I don't read pops up every time I stick in my flash drive, make it go away! whiiiine"


Log in to reply