Strange behaviour from new hard drive

Choonster

I recently bought a 4 TB WD Blue HDD, but I'm seeing some strange behaviour where there seems to be less space on the drive than there should be.

I purchased this drive as a replacement for an existing 3 TB one that I'm using to store a bunch of videos. I'd copied over most of the existing videos, but then when I tried to copy over a 1.04 TB archive it failed because there wasn't enough space; despite there being something like 1.5 TB left on the drive.

After that I decided to reformat it (this time I deselected "Perform a quick format") and copy over the archive by itself. Now that the archive has been copied, Windows says that 2.09 TB of space has been used on the drive; despite the file only being 1.04 TB. WinDirStat only shows 1.0 TB of space being used on the drive, from the archive and a couple of small system files.

This is the Properties window for the drive:

This is WinDirStat's report for the drive:

Does anyone know what could be going on here?

I've heard of scams with fake hard drives/flash drives that trick the system into thinking that they're much larger than they actually are, but this is a reputable brand from a legitimate site (I've bought a lot of stuff from them before); so I wouldn't expect to be scammed here.

PleegWat

@Choonster You have drive compression enabled. Does that cause windows to fuzz up the numbers for 'expected uncompressed' capacity?
Certainly it's going to be useless if you're going to be storing files which are already compressed.

Choonster

@PleegWat I'll try disabling drive compression.

It's not all archives, there are some regular videos as well.

PleegWat

@Choonster said in Strange behaviour from new hard drive:

@PleegWat I'll try disabling drive compression.

It's not all archives, there are some regular videos as well.

Which almost certainly already have lossy compression applied. Files with lossy compression rarely benefit from additional lossless compression, if the lossy compression format was well-designed.

Choonster

@PleegWat said in Strange behaviour from new hard drive:

@Choonster said in Strange behaviour from new hard drive:

@PleegWat I'll try disabling drive compression.

It's not all archives, there are some regular videos as well.

Which almost certainly already have lossy compression applied. Files with lossy compression rarely benefit from additional lossless compression, if the lossy compression format was well-designed.

They're fairly standard MP4 files, but the 7z compression does help a little bit; e.g. one archive is about 30 GB unpacked or 26 GB packed.

Tsaukpaetra

@Choonster said in Strange behaviour from new hard drive:

a bunch of videos.

At first I was thinking maybe you set the allocation unit size too large and had a node_modules folder in there, but now I'm not so sure.

Just in case, can you do a fsutil fsinfo ntfsinfo h: (elevated, natch) and post the result?

It should look something like this:

It should be 4Kb where it says Bytes Per Cluster, but if it's mostly huge files you'd technically be (slightly) better served with larger.

Tsaukpaetra

@Choonster said in Strange behaviour from new hard drive:

there wasn't enough space;

Ooh, one more thing, can you check the fragmentation? Windows might be doing a dumb and assuming that the file needs to be contiguous and you're hitting a case where there is no contiguous block of space that large (most likely because the MFT would intersect it)?

acrow

So, Windows is reporting the amount of space consumed as double that of the actual size of the file?
To me, this sounds like Windows wants to reserve extra space on the drive (for something), equivalent to the size of the largest file on the drive.

Maybe for storing Undo-history? Like, Volume Shadow Copy or something?

acrow

May also be related to a System Restore Point, according to:

Kevin Arrows / Jun 16, 2019

How to Fix Windows Reporting Wrong Hard Disk Free Space

Several users have been approaching us with questions after noticing that Windows is reporting wrong free space. The majority of users have discovered

.

Choonster

@Tsaukpaetra fsutil output:

NTFS Volume Serial Number :        0x44dad159dad14836
NTFS Version      :                3.1
LFS Version       :                2.0
Total Sectors     :                7,814,000,639  (  3.6 TB)
Total Clusters    :                  976,750,079  (  3.6 TB)
Free Clusters     :                  198,255,229  (756.3 GB)
Total Reserved Clusters :            156,012,327  (595.1 GB)
Reserved For Storage Reserve :                 0  (  0.0 KB)
Bytes Per Sector  :                512
Bytes Per Physical Sector :        4096
Bytes Per Cluster :                4096
Bytes Per FileRecord Segment    :  1024
Clusters Per FileRecord Segment :  0
Mft Valid Data Length :            7.00 MB
Mft Start Lcn  :                   0x00000000000c0000
Mft2 Start Lcn :                   0x0000000000000002
Mft Zone Start :                   0x00000000000c0000
Mft Zone End   :                   0x00000000000cc820
MFT Zone Size  :                   200.13 MB
Max Device Trim Extent Count :     0
Max Device Trim Byte Count :       0
Max Volume Trim Extent Count :     62
Max Volume Trim Byte Count :       0x40000000
Resource Manager Identifier :      7ED776D3-0A80-11EB-80C8-001A7DDA7113

I just used the default formatting settings, looks like Bytes Per Cluster is 4 KB.

Not entirely sure how to check the fragmentation.

I've been robocopying the rest of the videos/archives onto the new drive since last night, which has reserved the space for those files in advance; but the used space seems to be growing slowly and the numbers still don't quite add up. I'll see if the copy ends up completing or not.

Choonster

@Choonster WinDirStat is showing 2.9 TB of used space, but Windows is showing 3.53 (which is growing slowly); so there's definitely still a discrepency.

Choonster

@acrow Unfortunately none of those seem to be the issue.

acrow

@Choonster said in Strange behaviour from new hard drive:

@Choonster WinDirStat is showing 2.9 TB of used space, but Windows is showing 3.53 (which is growing slowly); so there's definitely still a discrepency.

Well, the growing part kinda rules out any issues with drive formatting or manufacturer error. So it's definitely Windows shenanigans. Unless you have a cryptolocker virus.

Just to check, you're not running 1803, are you?

Crazy theory: Maybe Windows is storing update files there, and hiding them from sight? So it can spread them P2P style?

Choonster

@acrow said in Strange behaviour from new hard drive:

@Choonster said in Strange behaviour from new hard drive:

@Choonster WinDirStat is showing 2.9 TB of used space, but Windows is showing 3.53 (which is growing slowly); so there's definitely still a discrepency.

Well, the growing part kinda rules out any issues with drive formatting or manufacturer error. So it's definitely Windows shenanigans. Unless you have a cryptolocker virus.

Just to check, you're not running 1803, are you?

Crazy theory: Maybe Windows is storing update files there, and hiding them from sight? So it can spread them P2P style?

No, I'm not running 1803.

I certainly hope I don't have a cryptolocker virus.

Choonster

@Choonster Now WinDirStat is showing 3.0 TB and Windows is showing 3.15 TB (down from 3.53 TB). It should be about 3.27 TB, so it's looking a bit more accurate now.

Unperverted Vixen

Could this be an SMR drive? I don't know how the caches in those work, but if they're considered part of the normal drive space, then the drive would need to allocate space for new files in both the cache and destination regions. So a new file would show up as using twice the space required, until the migration from cache to destination could complete.

Tsaukpaetra

@Unperverted-Vixen said in Strange behaviour from new hard drive:

Could this be an SMR drive? I don't know how the caches in those work, but if they're considered part of the normal drive space, then the drive would need to allocate space for new files in both the cache and destination regions. So a new file would show up as using twice the space required, until the migration from cache to destination could complete.

That would be interesting. As far as I know, most SMR drives don't expose that property to the old OS (which is why the whole hubbub about ZFS happened).

PleegWat

@Choonster said in Strange behaviour from new hard drive:

@Choonster WinDirStat is showing 2.9 TB of used space, but Windows is showing 3.53 (which is growing slowly); so there's definitely still a discrepency.

Note that about matches your Total Reserved Clusters size in the post right above that.

dkf

@Choonster said in Strange behaviour from new hard drive:

Total Reserved Clusters :            156,012,327  (595.1 GB)

Hello there…

Choonster

@Unperverted-Vixen said in Strange behaviour from new hard drive:

Could this be an SMR drive? I don't know how the caches in those work, but if they're considered part of the normal drive space, then the drive would need to allocate space for new files in both the cache and destination regions. So a new file would show up as using twice the space required, until the migration from cache to destination could complete.

According to this site, the EZRZ drive I bought is CMR; the SMR model is EZAZ instead.

Choonster

@Choonster robocopy has copied as much as it can, but it thinks it's run out of disk space. WinDirStat is showing 3.3 TB and Windows is showing 3.56 TB in use.

When I enable "Show Free Space" and "Show Unknown" in WinDirStat (which I only just discovered), the <Free Space> entry matches what's shown by Windows and the <Unknown> entry matches the discrepancy between the total file size and the used space.

The <Unkown> entry also matches the "Total Reserved Clusters" figure from fsutil:

NTFS Volume Serial Number :        0x44dad159dad14836
NTFS Version      :                3.1
LFS Version       :                2.0
Total Sectors     :                7,814,000,639  (  3.6 TB)
Total Clusters    :                  976,750,079  (  3.6 TB)
Free Clusters     :                   83,108,129  (317.0 GB)
Total Reserved Clusters :             64,373,998  (245.6 GB)
Reserved For Storage Reserve :                 0  (  0.0 KB)
Bytes Per Sector  :                512
Bytes Per Physical Sector :        4096
Bytes Per Cluster :                4096
Bytes Per FileRecord Segment    :  1024
Clusters Per FileRecord Segment :  0
Mft Valid Data Length :            13.75 MB
Mft Start Lcn  :                   0x00000000000c0000
Mft2 Start Lcn :                   0x0000000000000002
Mft Zone Start :                   0x00000000000c0000
Mft Zone End   :                   0x00000000000cc820
MFT Zone Size  :                   200.13 MB
Max Device Trim Extent Count :     0
Max Device Trim Byte Count :       0
Max Volume Trim Extent Count :     62
Max Volume Trim Byte Count :       0x40000000
Resource Manager Identifier :      7ED776D3-0A80-11EB-80C8-001A7DDA7113

I still have no idea what's using this space, but I'm going to try enabling and then disabling compression for all the files on the drive (most of them had the "compressed" flag copied from the old drive).

Tsaukpaetra

@Choonster said in Strange behaviour from new hard drive:

I still have no idea what's using this space

Not even Windows knows!

slapout1

@Choonster Is the archive one large file or multiple files?

cvi

@Choonster said in Strange behaviour from new hard drive:

I still have no idea what's using this space, but I'm going to try enabling and then disabling compression for all the files on the drive (most of them had the "compressed" flag copied from the old drive).

Could it be that this is some sort of staging area used for the compression? I.e., compression is expensive, and happens slowly in the background (and for some reason, Windows keeps the original data around until compression succeeds)?

Does this change if you leave the PC idle for a few hours?

Choonster

@cvi said in Strange behaviour from new hard drive:

@Choonster said in Strange behaviour from new hard drive:

I still have no idea what's using this space, but I'm going to try enabling and then disabling compression for all the files on the drive (most of them had the "compressed" flag copied from the old drive).

Could it be that this is some sort of staging area used for the compression? I.e., compression is expensive, and happens slowly in the background (and for some reason, Windows keeps the original data around until compression succeeds)?

Does this change if you leave the PC idle for a few hours?

It could be something like this.

I used the PowerShell script below to enable compression for the whole drive a day or two ago, and when I checked fsutil recently the Total Reserved Clusters figure was down to about 3 GB. I ran robocopy again after and it tried and failed to copy a new file to the drive. When I checked fsutil again just now, Total Reserved Clusters was down to 51.2 MB.

So some combination of enabling compression and leaving it idle seems to have freed up this missing space.

 Get-ChildItem H:\ -Recurse |
  ForEach-Object {
      compact /C $_.FullName
  }

Luck Man

@Choonster said in Strange behaviour from new hard drive:

@cvi said in Strange behaviour from new hard drive:

@Choonster said in Strange behaviour from new hard drive:

I still have no idea what's using this space, but I'm going to try enabling and then disabling compression for all the files on the drive (most of them had the "compressed" flag copied from the old drive).

Could it be that this is some sort of staging area used for the compression? I.e., compression is expensive, and happens slowly in the background (and for some reason, Windows keeps the original data around until compression succeeds)?

Does this change if you leave the PC idle for a few hours?

It could be something like this.

I used the PowerShell script below to enable compression for the whole drive a day or two ago, and when I checked fsutil recently the Total Reserved Clusters figure was down to about 3 GB. I ran robocopy again after and it tried and failed to copy a new file to the drive. When I checked fsutil again just now, Total Reserved Clusters was down to 51.2 MB.

So some combination of enabling compression and leaving it idle seems to have freed up this missing space.
 Get-ChildItem H:\ -Recurse |
  ForEach-Object {
      compact /C $_.FullName
  }

Hi, I just bought a new WD My Book 4TB, and I also encounter this Total Reserved Clusters problem. Although, mine is not as big as yours, only 4.7 GB, currently.
Did you finally find a solution to this problem?
Can you let me know?

error

Please send me the codes as well.

Gribnit

@error said in Strange behaviour from new hard drive:

Please send me the codes as well.

I also this problem. Blocked this past sprint. Revert with the needful.

Choonster

@Luck-Man The post you quoted was the closest I came to a solution.

Luck Man

@Choonster said in Strange behaviour from new hard drive:

@Luck-Man The post you quoted was the closest I came to a solution.

I've never tried enabling compression before.
Did it take a long time?
I'm going to move about 3.4 TB into it.

Choonster

@Luck-Man said in Strange behaviour from new hard drive:

@Choonster said in Strange behaviour from new hard drive:

@Luck-Man The post you quoted was the closest I came to a solution.

I've never tried enabling compression before.
Did it take a long time?
I'm going to move about 3.4 TB into it.

I don't remember it that much, but I think it did take quite a while.