Is ExFAT slower than NTFS?



  • I'm contemplating moving my storage disks to ExFAT, to better facilitate using them from both Windows and Mac.

    So I formatted a fresh drive to ExFAT and copied files from one of my old partitions over. Then I noticed something strange. ExFAT seems to be noticeably slower than NTFS at just stating the file and directory metadata. Look at the screenshot below.

    Both property sheets were opened at roughly the same time. NTFS drive (left) has already finished counting the files, while ExFAT drive (right) is only 1/3 of the way in.

    This is repeatable every time I try it, on different directories. I did a quick copy test, but while speeds were all over the place, there was no indication NTFS is many times faster than ExFAT. The disparity seems to be limited to random disk access.

    Both drives are the same model, plugged in the same motherboard. Both are GPT, cluster size 4kb. NTFS drive has an extra partition that plays no role here.

    Before I dig any further, a sanity check please. Am I missing something? Is there anything wrong here at all, or is this expected with ExFAT?


  • Impossible Mission Players - A

    @cartman82 said:

    Am I missing something?

    Disk caching? Windows will keep stuff in memory (including, presumably, file tables) and if you've recently scanned the tree, subsequent accesses should be faster.

    I'm sure there are filesystem benchmarking wares you could use to test.



  • @Tsaukpaetra said:

    Disk caching? Windows will keep stuff in memory (including, presumably, file tables) and if you've recently scanned the tree, subsequent accesses should be faster.

    I'm sure there are filesystem benchmarking wares you could use to test.

    Repeated tests yield the same results.

    I'm not sure what would a benchmark show me. When the difference is x2 - x3, I can see it with my own eyes.


  • SockDev

    ExFAT will look slower than NTFS because NTFS assumes it's on a permanent disc and so can take its time to actually write data blocks because you won't pull the plug on it. ExFAT makes the opposite assumption.

    IIRC there's a setting that tells an NTFS volume it's a removable one so windows will stop writecaching the volume, but bugger if i can remember where that setting is......


  • BINNED

    Why write caching should affect reading directory meta data?

    My guess is Microsoft has optimized for NTFS access but not for ExFAT (inset intentional MS conspiracy here).


  • Impossible Mission Players - A

    @cartman82 said:

    Repeated tests yield the same results.

    Interesting.

    @cartman82 said:

    I'm not sure what would a benchmark show me. When the difference is x2 - x3, I can see it with my own eyes.

    Well, some people have apparently done some benchmarks already.

    For the most part, it seems merely reading/writing files is roughly the same (or slightly worse with exFAT).

    Unless you're doing pretty intensive deadline I/O, I don't think it will matter too much in real usage scenarios. After all, how often are you counting all the file on the disk?

    Better test would have been to see how long it takes to copy files around (a la regular usage).



  • I've experienced problems under Windows with directories with a large number of entries. That slowed down the file system to no end. Can't remember which fs, though. Do you perhaps have very large directories?


  • area_pol

    Since you experience a significant difference on scanning the metadata but not on file read/write, it may be due to the different ways of storing/accessing metadata on those filesystems.



  • @accalia said:

    ExFAT will look slower than NTFS because NTFS assumes it's on a permanent disc and so can take its time to actually write data blocks because you won't pull the plug on it. ExFAT makes the opposite assumption.

    IIRC there's a setting that tells an NTFS volume it's a removable one so windows will stop writecaching the volume, but bugger if i can remember where that setting is......

    Interesting. Off to google if there are any registry switches controlling the behavior of ExFAT.

    @Tsaukpaetra said:

    Well, some people have apparently done some benchmarks already.

    For the most part, it seems merely reading/writing files is roughly the same (or slightly worse with exFAT).

    Unless you're doing pretty intensive deadline I/O, I don't think it will matter too much in real usage scenarios.

    Yeah, I've seen these. That's why the results are so confusing.

    I tried just copying a large file, and the speeds are comparable.

    @Hanzo said:

    I've experienced problems under Windows with directories with a large number of entries. That slowed down the file system to no end. Can't remember which fs, though. Do you perhaps have very large directories?

    It's exactly the same directory tree on both drives. Plenty of small files.

    @Adynathos said:

    Since you experience a significant difference on scanning the metadata but not on file read/write, it may be due to the different ways of storing/accessing metadata on those filesystems.

    Yeah. The question is, is it? And is there anything I can do about it? And why haven't the benchmarks online caught this?


  • Impossible Mission Players - A

    @cartman82 said:

    Yeah. The question is, is it?

    Eh, I'm going to say, "probably" and just be content that it even works cross platform. Now, if only we could test on the alter-OS you formatted the disk to be cross-compatible for...

    @cartman82 said:

    And is there anything I can do about it?
    Unlikely. Unless there's a modern version of fastdisk.exe available...

    @cartman82 said:

    And why haven't the benchmarks online caught this?
    Unique usage situation. Apparently people are more concerned with File I/O than directory reads.



  • Doesn't NTFS use a radically different directory structure than FAT?

    I do know that recovering and/or repairing a corrupted file system is far easier with FAT than with NTFS.

    Also, FAT won't give you weird permission headaches.


  • Winner of the 2016 Presidential Election

    @Adynathos said:

    Since you experience a significant difference on scanning the metadata but not on file read/write, it may be due to the different ways of storing/accessing metadata on those filesystems.

    @cartman82 said:

    Yeah. The question is, is it?

    Seems to be. From wikipedia:

    exFAT and the rest of the FAT family of filesystems does not use indexes for filenames, unlike NTFS which uses B-trees for file searching. When a file is accessed, the directory must be sequentially searched until a match is found.

    This appears to be confirmed in other places:

    The FAT file system uses a simple linked-list arrangement for storing large directories: the first few files are listed in the first cluster of the directory, and then the next files go into the next cluster, which is linked to the first, and so on. This is simple to implement, but means that every time you look at the directory you must scan it from start to end and then sort it for presentation to the user. It also makes it time-consuming to locate individual files in the index, especially with very large directories.
    To improve performance, NTFS directories use a special data management structure called a B-tree. [⋮] From a practical standpoint, the use of B-trees means that the directories are essentially "self-sorting". There is a bit more overhead involved when adding files to an NTFS directory, because they must be placed in this special structure. However, the payoff occurs when the directories are used. The time required to find a particular file under NTFS is dramatically reduced compared to an unsorted linked-list structure--especially for very large directories.

    That's talking about FAT rather than exFAT, but other documents lead me to believe the important parts aren't incorrect. (This presentation also indicates that exFAT stores file names as a set of up to 17 15-character "File Name Extension Directory Entr[ies]")

    So it seems that what you're seeing is the performance difference of a B-tree based FS (NTFS) vs. a table + linked-list based one (exFAT).

    @cartman82 said:

    And is there anything I can do about it?

    Probably not, since it seems intrinsic in the design.

    @cartman82 said:

    And why haven't the benchmarks online caught this?

    ¯\_(ツ)_/¯



  • @cartman82 said:

    It's exactly the same directory tree on both drives.

    I got that part. But as the other post suggests, it can be a problem to have large directories. Try making them smaller by adding an extra layer of directories.



  • @cartman82 said:

    is this expected with ExFAT?

    If ExFAT works much like FAT, it will end up with little directory files scattered all over the place. NTFS keeps all the directories inside the Master File Table, which it goes to some lengths to keep contiguous and optimally placed. I would not be at all surprised to learn that operations involving recursive directory tree walking run a lot faster on NTFS than on any variety of FAT.



  • Nobody yet gave the obvious reply?

    ExFAT is slower because it's FAT!

    undefined



  • @cartman82 said:

    I'm contemplating moving my storage disks to ExFAT, to better facilitate using them from both Windows and Mac.

    OSX doesn't support NTFS?


  • SockDev

    Boot Camp's implementation of NTFS is read only.


  • SockDev

    Not to mention that last I knew, any non-MS implementation that supports writes either only supports ancient versions of NTFS, or are so buggy they're not useable.


  • area_pol

    @RaceProUK said:

    Not to mention that last I knew, any non-MS implementation that supports writes either only supports ancient versions of NTFS, or are so buggy they're not useable.

    I must disagree, I read and write NTFS partitions from Linux and have not encountered any problems apart from being unable to write if Windows is hibernated.


  • SockDev

    Well then, it seems the situation has improved since I last looked 😄


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.