Big FAT32 Hard Drives
-
I connected a brand-new 500 gigabyte Western Digital "My Book" USB hard drive to the computer. Then I started a backup to the drive.
It failed after an hour: "End of Media encountered while backing up to non-removable media." The backup file was 4 gigabytes in size.
I wonder why the drive can't hold a file bigger than 4 gigabytes?
Maybe because it is pre-formatted as ... FAT32?
-
Of course it is FAT32. That's because that is what is most likely to work. It works with NT, it works with the DOSoid Windows (ME, 98, even 95 if it is the version with USB support), it works with MacOS and it works with Linux, all without needing to format.
Now in your particular case, FAT32 isn't the right choice, so format it, already.
The real WTF is that you are complaining about something that makes this device more accessible to more users. I think the 80/20 rule applies here.
-
@Critter said:
Of course it is FAT32. That's because that is what is most likely to work. It works with NT, it works with the DOSoid Windows (ME, 98, even 95 if it is the version with USB support), it works with MacOS and it works with Linux, all without needing to format.
Now in your particular case, FAT32 isn't the right choice, so format it, already.
The real WTF is that you are complaining about something that makes this device more accessible to more users. I think the 80/20 rule applies here.
While I agree the 80/20 rules does apply, I don't think you are correct in your assessment that it will work under all those systems. I do believe that it will fail in this case under all those system for the reason he states.
Now my question is this, barring the single file of 4gig could he have create 5 files a gig in size each and have it work? What was the size limit onFAT32 or was it simply sector size which USB devices ignore? If this test failed then yes, having it formatted for FAT32 is a WTF because by default it would not support the 500gig size.
-
I'm always suspicious when hard drives come pre-formatted. I've bought 3 WD hard drives, none of them were formatted. If you didn't buy from a dealer with a solid reputation (however you want to define that), you should check the drive for previous data.
Could be that someone started copying their pr0n onto the drive, had some problem with it, and brought it back to the dealer where it was formatted and re-shelved.
- Eam
Edit: I bought SATA drives, so the USB drives may be targeted at less-awesome customers and come with shitty formatting for a legitimate reason.
-
File size limit is approximately 4GB. According to http://www.allensmith.net/Storage/HDDlimit/FAT32.htm, Windows 2000/XP can not format FAT32 over 32GB (but CAN use drives formatted FAT32 that are over 32GB), whereas Windows 98 could format FAT32 up to about 128GB. Windows ME had a theoretical limit of about 4TB using FAT32.
-
@KattMan said:
Now my question is this, barring the single file of 4gig could he have create 5 files a gig in size each and have it work?
Absolutely, yes. He could even create five files of 4GB each. He just could not exceed 4GB per file.
What was the size limit onFAT32 or was it simply sector size which USB devices ignore? If this test failed then yes, having it formatted for FAT32 is a WTF because by default it would not support the 500gig size.
No, it hasn't got anything to do with USB. FAT has (as I mentioned a second ago) a 4GB file size limit, and the filesystem size is accommodated by changing the block size, if I recall correctly. While I don't have any first-hand experience that says FAT works at 500GB, I do have some that says it works at more than 4GB, 60GB to be specific.
FAT32 has one other annoying limitation, and that is 127 objects maximum in the root directory. This limit does not affect subdirectories. This is sometimes an obstacle with my MP3 player.
-
I have to call this not a WTF. The WD MyBook is basically a user friendly (read dead easy to use) external drive, designed for the general populace to plug in and do backups. It comes with some autorun backup programs for Windows, but works just fine on my Linux boxes as well (in case you couldn't tell I have one).
As already stated the 4GB limit is only for single files. The only time you should really have a problem is if you are doing video archiving, or personal video editing. Even then you should usually be below the 4GB limit.
-
As the original poster found out, the software they usually come with, Dantz (EMC?) Retrospect, won't split files into 4gb partitions. So if you just say "back it all up!" the first files it creates will be as large as your (compressed) files - overflowing 4gb.
-
Doesn't FAT32 have major slack space issues with partitions that big?
-
The main problem is that the only filesystem supported by Windows, Mac AND Linux is FAT32. If you format it as, say, ReiserFS, then it is only readable on Linux.
If you use NTFS, you can read/write to it on Windows and read-only on Linux (can Macs at least read NTFS?)
So you don't really have much option if you want compatibility across Win/Mac/Linux.
-
EXT2 or EXT3 might be a good choice. They are backwards/forwards compatible and support is built into linux. Fairly good EXT2 drivers (which are of course compatible with EXT3) exist for Windows and Mac as well.
-
@H3SO5 said:
If you use NTFS, ... read-only on Linux (can Macs at least read NTFS?)
This is no longer true.
-
@mthamil said:
@H3SO5 said:
If you use NTFS, ... read-only on Linux (can Macs at least read NTFS?)
This is no longer true.links or it didn't happen.
I was under the impression that, while you could theoretically write NTFS on linux, it was currently in pre-1.0 and known to occasionally break.
-
Have you seen the NTFS-3G driver ( http://www.ntfs-3g.org/ ) - it claims that "Most POSIX file system operations are supported, with the exception of full file ownership and access right support. ".
-
@Irrelevant said:
@mthamil said:
@H3SO5 said:
If you use NTFS, ... read-only on Linux (can Macs at least read NTFS?)
This is no longer true.links or it didn't happen.
I was under the impression that, while you could theoretically write NTFS on linux, it was currently in pre-1.0 and known to occasionally break.
It was just released as stable a few months ago, so I'm not surprised you didn't hear about it. It does have some performance issues - it hasn't been optimized. It isn't 100% as it will fail in some corner cases like extending the MFT in certain scenarios.
-
@mthamil said:
@H3SO5 said:
If you use NTFS, ... read-only on Linux (can Macs at least read NTFS?)
This is no longer true.
what version of what distro? Is it stable or dev?
-
@lpope187 said:
@Irrelevant said:
@mthamil said:
@H3SO5 said:
If you use NTFS, ... read-only on Linux (can Macs at least read NTFS?)
This is no longer true.links or it didn't happen.
I was under the impression that, while you could theoretically write NTFS on linux, it was currently in pre-1.0 and known to occasionally break.
It was just released as stable a few months ago, so I'm not surprised you didn't hear about it. It does have some performance issues - it hasn't been optimized. It isn't 100% as it will fail in some corner cases like extending the MFT in certain scenarios.
Thank you so much!
-
@bobday said:
I'm always suspicious when hard drives come pre-formatted. I've bought 3 WD hard drives, none of them were formatted. If you didn't buy from a dealer with a solid reputation (however you want to define that), you should check the drive for previous data.
Could be that someone started copying their pr0n onto the drive, had some problem with it, and brought it back to the dealer where it was formatted and re-shelved.
- Eam
Edit: I bought SATA drives, so the USB drives may be targeted at less-awesome customers and come with shitty formatting for a legitimate reason.
No, the USB Harddrives always seem to come pre-formatted as FAT 32.
-
@shadowman said:
No, the USB Harddrives always seem to come pre-formatted as FAT 32.
It's the only filesystem supported by windows by default that won't enthusiastically corrupt itself if you unplug the drive without unmounting it first (NTFS is notoriously bad at this).
It's also a terrible filesystem. Go with ext3 wherever possible.
-
great to hear about that NTSF thingy :)
I have a 300GB external maxtor usb drive methingy, and was getting sick of FAT32 as well.
Now i can finally transport those huge ummm linux dvd iso's on it.Currently it's holding around about 250GB of anime :P
It's weird how it doesn't really matter how much space you have, it will always find a way to fill up.
When i first bought my drive i though i would be able to keep free space for atleast a year, about 2 months later is was 95% full :)
-
@stratos said:
great to hear about that NTSF thingy :)
I have a 300GB external maxtor usb drive methingy, and was getting sick of FAT32 as well.
Now i can finally transport those huge ummm linux dvd iso's on it.Currently it's holding around about 250GB of anime :P
It's weird how it doesn't really matter how much space you have, it will always find a way to fill up.
When i first bought my drive i though i would be able to keep free space for atleast a year, about 2 months later is was 95% full :)
It's easy to fill drives when you download anime and porn all day.
-
@asuffield said:
It's the only filesystem supported by windows by default that won't enthusiastically corrupt itself if you unplug the drive without unmounting it first (NTFS is notoriously bad at this).
I would say that any journaling filesystem set to write to disk only when buffer is full on a USB connection is an error waiting to happen.
@asuffield said:
It's also a terrible filesystem. Go with ext3 wherever possible.
Could you explain why you think NTFS is so bad?
-
@bobday said:
Could be that someone started copying their pr0n onto the drive, had some problem with it, and brought it back to the dealer where it was formatted and re-shelved.
Are you suggesting it's a bad thing to get somebody else's porn?
I mean, granted, the harddrive might not work. But free porn!
-
@smbell said:
I have to call this not a WTF. The WD MyBook is basically a user friendly (read dead easy to use) external drive, designed for the general populace to plug in and do backups. It comes with some autorun backup programs for Windows, but works just fine on my Linux boxes as well (in case you couldn't tell I have one).
As already stated the 4GB limit is only for single files. The only time you should really have a problem is if you are doing video archiving, or personal video editing. Even then you should usually be below the 4GB limit.
i'm pleased to see that you've also started tagging your posts with "tag abuse"...warm fuzzies
-
@newfweiler said:
I connected a brand-new 500 gigabyte Western Digital "My Book" USB hard drive to the computer. Then I started a backup to the drive.
It failed after an hour: "End of Media encountered while backing up to non-removable media." The backup file was 4 gigabytes in size.
I wonder why the drive can't hold a file bigger than 4 gigabytes?
Maybe because it is pre-formatted as ... FAT32?
Of course I reformatted it before posting. I used NTFS for convenience.
In my opinion, everything I buy should come out of the box as one of the following:
1. Some preparation required, or
2. No preparation required.
It should either be completely ready to use, or not ready to use. Either just plug it in and it works, or attach the power cord, pull out the battery tab, format it and it works. One or the other, please.
The disk came preformatted with FAT32 because that's the only format that can be read by almost any system. At the same time, FAT32 on a 500GB drive is of no use to anyone because of the file size limitation.
The only reason the disk comes formatted at all is so that it can install the Google Toolbar and show you a few ads when you plug it in. For Your Convenience of course. Oh, yes, and so it can install some hard drive utilities you don't need.
-
I've been successfully using NTFS-3G on my intel macbook to access an external 300 gig drive for several months now. There's a package out there somewhere that makes installation as simple as possible -- run an installer, and you're done. It may be a touch on the slow side, but since I only store bulk data on there, it doesn't really matter.
-
@Saxov said:
@asuffield said:
It's the only filesystem supported by windows by default that won't enthusiastically corrupt itself if you unplug the drive without unmounting it first (NTFS is notoriously bad at this).
I would say that any journaling filesystem set to write to disk only when buffer is full on a USB connection is an error waiting to happen.
NTFS only uses metadata journalling anyway - so after running chkdsk, all the directories and file sizes will be correct, but some of the files will contain garbage. It is entirely wrong for removable media.
@asuffield said:
It's also a terrible filesystem. Go with ext3 wherever possible.
Could you explain why you think NTFS is so bad?Aside from the above inappropriateness for removable media? It uses space inefficiently, you have to run a defragmenter over the thing on a regular basis just to clean up the mess made by the insane allocation strategy, it can't be reliably used on platforms other than windows, and as filesystems go, it's amazingly slow. That's off the top of my head and without even considering the whacked-out on-disk format (as a result of inheritance from HPFS and then kludging features on top).
-
@seaturnip said:
Doesn't FAT32 have major slack space issues with partitions that big?
Spectacularly, yes. I recently changed an 80GB FAT32 partition from 32-kb clusters (how Windows formatted it) to 16-kb clusters (how Linux formatted it), and saved 5GB. I estimate I could save another 4GB by changing from FAT32 to the 2-kb blocks of ext2, but then the partition would no longer be natively readable under Win98SE or WinXP.
-
@asuffield said:
It's the only filesystem supported by windows by default that won't enthusiastically corrupt itself if you unplug the drive without unmounting it first (NTFS is notoriously bad at this).
It's also a terrible filesystem. Go with ext3 wherever possible.
I'm certain it's the other way around. NTFS is quite good at not corrupting itself. Usually whenever a computer with a FAT32 drive crashes Windows will have to run chkdsk to pick up the pieces. With NTFS all you usually end up with are some weird timestamps occasionally.
Only problem I've had on NTFS is once or twice I "safely remove hardware" and then get a warning message "Delayed write to X:$mft failed" (That's the master file table! ACK!) But that's hardly NTFS' fault, since Windows itself was supposed to be sure everything was finished before dismounting the drive.
Also I've found the ext2 drivers for windows tend to require ubuntu to repair the filesystem on the next boot... :/ that can't be good. I haven't experienced any real problems from it though. Oh yeah except windows dumps a recycled folder onto my ubuntu partiton.
I formatted both my thumb drives as NTFS because I don't really interact with non-Windows boxes a lot. I formatted all of my SD cards as FAT32 (then checked to make sure my devices could still use them). I also used the smallest cluster size possible (this reduces file "slack" as one previous poster put it). I've noted that Vista will cryptically say "Unable to complete the format." if you've set the cluster size too small for the disk size. Just increase it one notch up and try again.
-
@The MAZZTer said:
@asuffield said:
It's the only filesystem supported by windows by default that won't enthusiastically corrupt itself if you unplug the drive without unmounting it first (NTFS is notoriously bad at this).
It's also a terrible filesystem. Go with ext3 wherever possible.
I'm certain it's the other way around. NTFS is quite good at not corrupting itself. Usually whenever a computer with a FAT32 drive crashes Windows will have to run chkdsk to pick up the pieces. With NTFS all you usually end up with are some weird timestamps occasionally.
Absence of errors is not absence of corruption. NTFS corruption is, indeed, usually silent. You don't notice until six months later, when you go to open that critical file and find it's toast. Furthermore, it's data corruption rather than metadata corruption, so chkdsk is indeed not involved (chkdsk is pure metadata stuff). Yes, FAT32 corruption is more obvious, but it's the journalling design in NTFS that makes it so susceptible to corruption when interrupted. You have to realise that the current version of NTFS was designed with the philosophy: "Some data loss is okay, so long as we never bother the user with chkdsk or error messages", and this is precisely what you are observing.
Only problem I've had on NTFS is once or twice I "safely remove hardware" and then get a warning message "Delayed write to X:$mft failed" (That's the master file table! ACK!) But that's hardly NTFS' fault, since Windows itself was supposed to be sure everything was finished before dismounting the drive.
Really can't imagine why you think there is a distinction between NTFS and the Windows filesystem layer. Same project, same developers. Only difference is which source file the code happened to be in.
It's none of the above though. This particular error is nonsense - the only problem here is a bogus error message when no error occurred. I vaguely recall tracking it down to a bug in one of the drivers shipped with windows, and locating an update that eliminates it, but that was a few years ago and I really can't remember which one. I'd be surprised it was still around, except that I'm used to Microsoft's policy on driver bugs ("blame the vendors and let them ship an update").
Also I've found the ext2 drivers for windows tend to require ubuntu to repair the filesystem on the next boot... :/ that can't be good. I haven't experienced any real problems from it though.
It's not a real problem, per se. Unlike NTFS, the ext2 drivers are written by people who really really care about the safety of your data. There's a flag in the filesystem that records whether or not the driver is sure that the filesystem is intact. At any point when the filesystem is mounted read-write, the flag is set to "dirty". It is only set back to "clean" when the filesystem is unmounted safely, usually when the computer is rebooted. If this does not go perfectly, it'll remain set to "dirty", and on the next boot to a unix platform it'll get fsck run over it just to make sure. The Windows drivers have some kind of issue with setting the flag to "clean" on reboot, if you don't manually unmount the filesystem first (it's some kind of obscure problem with the order in which Windows closes programs while shutting down) - but there's nothing actually wrong with the filesystem, it's just the flag that's left behind. Telling the ext2 driver to unmount before you reboot should stop it from happening.
-
@lpope187 said:
It was just released as stable a few months ago, so I'm not surprised you didn't hear about it. It does have some performance issues - it hasn't been optimized. It isn't 100% as it will fail in some corner cases like extending the MFT in certain scenarios.
OK, I stand corrected. Didn't remember it because I haven't had an NTFS partition for almost a year.
