Windows Server Storage Spaces
-
In a completely unrelated but similar vein as the thread about Ceph, does anyone have experience running Storage Spaces in environments that are write heavy? Over the past couple of years we have become pretty heavily involved in installing DVRs and PoE IP cameras. Now we are at the point of having clients that are needing to upgrade their storage so I am reinvestigating our build outs for NVR servers.
Windows Server Storage Spaces in theory seems like a good option. In theory. Pass the disks through directly to the OS, make a Storage Space out of them, in the future if we need to expand storage we just add more disks and import them in to the Storage Space and rebalance.
Reading about the experience of others, it seems like write performance is completely atrocious. DVRs do a lot of writing and deleting. That's about it. Reading happens extremely infrequently. Like, every few weeks someone might read off 30-100MB and that's about it. But the volume cannot be a bottleneck for writes. We have one client that is currently running 40 cameras on a 16TB storage volume and we are about to expand their office space by 50% and probably another 30 cameras so we will need to expand the storage. The last time we did so we burned the existing volume and replaced it with four 8TB drives in a RAID10. This time if at all possible I would like to move to something that will allow us to expand the volume without removing the existing one, and RAID controllers just don't support such things.
The current server is a 4 bay R710 chassis (they had it on hand and it wasn't doing anything so we repurposed it for this). We will probably move them to some manner of 12 bay chassis this time. In theory we could use the RAID controller to create new RAID volumes each time and then merge them at the OS level in to a JBOD of RAID arrays, but that solution seems inelegant. Individual drives could not be upgraded, it would be all or nothing.
The other option I considered was to pass all of the disks through to the OS and use something like FlexRAID. Or, since the DVR software supports storage to NAS using something like a Drobo, but that solution also seems inelegant but would be better if we used iSCSI or something to mount the drive.
Is there another solution that I have missed? In an ideal world the solution would,
Have to be:
- Compatible with Windows
- Allow storage upgrades without destroying the array and rebuilding
- Have decent write performance
- Not be a total fucking nightmare to work with
- As simple as possible (some of these systems are turned loose to on-site admins that articles on this site are written about)
-
@Polygeekery said in Windows Server Storage Spaces:
Compatible with Windows
What does this mean, in context? Like "I can have raw access in Windows as if it were a physical drive"? Or "Windows can talk to it like a network share"?
-
@Tsaukpaetra ideally raw access, installed in the same set of hardware. I am open to other options though if they make sense for the use case. As I have said before I am a big fan of "Why?" questions and the only unacceptable answer is "Because we have always done it this way."
I think my ideal solution would be Windows Storage Spaces, except for when you search for "Windows Server Storage Spaces performance" you get tons of hits like this:
Microsoft Storage Spaces Is Hot Garbage For Parity Storage
I love parity storage. Whether it’s traditional RAID 5/6, erasure coding, raidz/raid2z, whatever. It gives you redundancy on your data without requiring double the drives that mirroring or mi…
https://foxdeploy.com/2015/10/30/windows-vs-intel-raid-performance-smackdown/
With this purpose being incredibly write heavy, I just don't think it will work. But some of these articles are older so fixes may have been made. The testing they have done is also synthetic testing which IME is a poor predictor of real world performance.
-
Watching this closely because I'm about to throw together an NVR system...
-
@Weng if you have any questions on the subject feel free to ask them. We have several deployed now so I might be able to help out.
-
@Polygeekery said in Windows Server Storage Spaces:
write heavy
Yeah, IMO most storage systems are optimized for reads.
Supposedly you can make the fast tier allocate more space for write cache in storage spaces, but I think if your average write rate exceeds the total slowest total write rate of your rust then it won't help anyways, as the wrote cache is mostly to catch bursts of writes waiting to trickle down to the rust.
-
@Tsaukpaetra said in Windows Server Storage Spaces:
Supposedly you can make the fast tier allocate more space for write cache in storage spaces, but I think if your average write rate exceeds the total slowest total write rate of your rust then it won't help anyways, as the wrote cache is mostly to catch bursts of writes waiting to trickle down to the rust.
DVR service is pretty steady state, especially with the way that this particular client wants it configured. We used to only store clips when changes occurred, but they had an employee that was convinced that it was missing things (it wasn't) so we changed everything to just record straight to disk 24/7/365, which meant that it cut our retention period roughly in half. Then they decided to add some parking lot cameras and we went with 4K models to make sure they could capture detail at distance, which cut the retention period even more.
I logged in to the server and checked, we are running ~10MB/s to the drives, so writes aren't really that bad. But every thing written has a delete that goes with it as free space policy deletes the oldest items. That in theory should not increase load all that much as it should just mark that space as able to be written over.
Side note: When we setup one of these in the beginning the guy I sent out to do the install turned on deduplication and once it came time to free up space Windows just completely shit itself to the point that we had to destroy the volume and rebuild. Windows dedupe does not deal well with such workloads, and it does not fail gracefully.
-
@Weng said in Windows Server Storage Spaces:
Watching this closely because I'm about to throw together an NVR system...
Is it for your home/shop? If so I would seriously consider just using Ubiquiti equipment for it. Their new Gen2 Cloud Key has all the NVR stuff already in it and I believe you can hook up external hard drives to expand available storage space, plus they give you remote access to cameras and recordings for free without a monthly charge.
-
@Polygeekery said in Windows Server Storage Spaces:
@Tsaukpaetra said in Windows Server Storage Spaces:
Supposedly you can make the fast tier allocate more space for write cache in storage spaces, but I think if your average write rate exceeds the total slowest total write rate of your rust then it won't help anyways, as the wrote cache is mostly to catch bursts of writes waiting to trickle down to the rust.
DVR service is pretty steady state, especially with the way that this particular client wants it configured. We used to only store clips when changes occurred, but they had an employee that was convinced that it was missing things (it wasn't) so we changed everything to just record straight to disk 24/7/365, which meant that it cut our retention period roughly in half. Then they decided to add some parking lot cameras and we went with 4K models to make sure they could capture detail at distance, which cut the retention period even more.
I logged in to the server and checked, we are running ~10MB/s to the drives, so writes aren't really that bad. But every thing written has a delete that goes with it as free space policy deletes the oldest items. That in theory should not increase load all that much as it should just mark that space as able to be written over.
Side note: When we setup one of these in the beginning the guy I sent out to do the install turned on deduplication and once it came time to free up space Windows just completely shit itself to the point that we had to destroy the volume and rebuild. Windows dedupe does not deal well with such workloads, and it does not fail gracefully.
For these kinds of systems aren't Xeons overkill? Or are these also transcoding servers?
-
@Polygeekery said in Windows Server Storage Spaces:
turned on deduplication
Lol yeah you almost never need this, and it most certainly doesn't work on video content.
-
@mikehurley said in Windows Server Storage Spaces:
@Polygeekery said in Windows Server Storage Spaces:
@Tsaukpaetra said in Windows Server Storage Spaces:
Supposedly you can make the fast tier allocate more space for write cache in storage spaces, but I think if your average write rate exceeds the total slowest total write rate of your rust then it won't help anyways, as the wrote cache is mostly to catch bursts of writes waiting to trickle down to the rust.
DVR service is pretty steady state, especially with the way that this particular client wants it configured. We used to only store clips when changes occurred, but they had an employee that was convinced that it was missing things (it wasn't) so we changed everything to just record straight to disk 24/7/365, which meant that it cut our retention period roughly in half. Then they decided to add some parking lot cameras and we went with 4K models to make sure they could capture detail at distance, which cut the retention period even more.
I logged in to the server and checked, we are running ~10MB/s to the drives, so writes aren't really that bad. But every thing written has a delete that goes with it as free space policy deletes the oldest items. That in theory should not increase load all that much as it should just mark that space as able to be written over.
Side note: When we setup one of these in the beginning the guy I sent out to do the install turned on deduplication and once it came time to free up space Windows just completely shit itself to the point that we had to destroy the volume and rebuild. Windows dedupe does not deal well with such workloads, and it does not fail gracefully.
For these kinds of systems aren't Xeons overkill? Or are these also transcoding servers?
I think he mentioned it was a repurposed unit.
-
@mikehurley said in Windows Server Storage Spaces:
For these kinds of systems aren't Xeons overkill? Or are these also transcoding servers?
Maybe, but it was there and available for repurpose. Also, it was initially doing transcoding to overlay timestamps and such, but we ended up ditching that in a prior revision of the system due to other reasons and ended up doing the overlays at the camera and then going direct to disk.
It all depends on how many cameras and what resolutions you are running. As you can see we are running 8 cores and are at ~30-40% CPU. But we are also running 30-40 cameras on that system at a minimum resolution of 720P.
-
@Tsaukpaetra said in Windows Server Storage Spaces:
@mikehurley said in Windows Server Storage Spaces:
@Polygeekery said in Windows Server Storage Spaces:
@Tsaukpaetra said in Windows Server Storage Spaces:
Supposedly you can make the fast tier allocate more space for write cache in storage spaces, but I think if your average write rate exceeds the total slowest total write rate of your rust then it won't help anyways, as the wrote cache is mostly to catch bursts of writes waiting to trickle down to the rust.
DVR service is pretty steady state, especially with the way that this particular client wants it configured. We used to only store clips when changes occurred, but they had an employee that was convinced that it was missing things (it wasn't) so we changed everything to just record straight to disk 24/7/365, which meant that it cut our retention period roughly in half. Then they decided to add some parking lot cameras and we went with 4K models to make sure they could capture detail at distance, which cut the retention period even more.
I logged in to the server and checked, we are running ~10MB/s to the drives, so writes aren't really that bad. But every thing written has a delete that goes with it as free space policy deletes the oldest items. That in theory should not increase load all that much as it should just mark that space as able to be written over.
Side note: When we setup one of these in the beginning the guy I sent out to do the install turned on deduplication and once it came time to free up space Windows just completely shit itself to the point that we had to destroy the volume and rebuild. Windows dedupe does not deal well with such workloads, and it does not fail gracefully.
For these kinds of systems aren't Xeons overkill? Or are these also transcoding servers?
I think he mentioned it was a repurposed unit.
Well, that and it fits in the rack neatly, and we need the core count for the workload. A quad core i7 would choke on this workload.
Actual servers with multiple sockets and Xeons provide an upgrade path for expansion. If we ever start running in to CPU contention we could always hop on eBay and order up a couple of X5680 processors off the used market and drop those in to get more headroom.
Also, used servers are cheap. For SMBs they rarely need latest and greatest. Markets are flooded with high spec machines that are only one generation behind that SMBs can get years worth of life out of for a fraction of what a new machine would cost.
-
@Polygeekery Are you looking to have redundancy on the data, or do you want it purely for storage pooling?
If all you want is pooling, then the articles you linked to don't really apply. They discuss the performance going to pot with parity disks in the storage space. The second one showed the numbers for pure storage pooling, as well as drive mirroring, and they seemed not too different from hardware RAID.
-
@Unperverted-Vixen well, ideally there would be some redundancy. Some equivalent of RAID6 if possible. Or an implementation of RAID4 would even be fine. RAID10 would also work but would not be nearly as efficient on storage. Any solution has to be able to be expanded easily and ideally we would be able to replace individual disks in the storage array to upgrade capacity without having to burn the entire array.
The last storage upgrade we did we had to yank out four 4TB disks in RAID10 and replace them with 8TB disks. In an ideal world I could add some larger disks and then as storage prices come down remove smaller disks from the array and have it rebuild and redistribute the data. Much like Drobos do.
-
@Polygeekery said in Windows Server Storage Spaces:
The last storage upgrade we did we had to yank out four 4TB disks in RAID10 and replace them with 8TB disks. In an ideal world I could add some larger disks and then as storage prices come down remove smaller disks from the array and have it rebuild and redistribute the data. Much like Drobos do.
I actually recently did that on my FreeNAS box recently. Just warm-swapped the old drive with a new one with higher capacity, and as soon as all drives in the VDEV (two drives per vdev, two vdevs) were at the higher capacity it instantly updated the pool to use the increased space. No idea if that's how it would work in Storage Spaces, but I assume it would be similar.
-
@Tsaukpaetra I guess I am going to have to load up a box and give it a test. I was really hoping that someone would have real world experience with it in some capacity.
The more I think about it, the more I wonder if a Drobo wouldn't be the way to go. I think I have a generation or two back iSCSI capable version I could give it a try on before I go spending $2K of the client's money on something that might not work properly.
-
Looking in to their product line, their desktop 5-bay DAS are compatible with Windows, but their 8-bay rackmountables are Apple only.
-
@Tsaukpaetra said in Windows Server Storage Spaces:
No idea if that's how it would work in Storage Spaces, but I assume it would be similar.
That's exactly how it works.
@Polygeekery said in Windows Server Storage Spaces:
I guess I am going to have to load up a box and give it a test. I was really hoping that someone would have real world experience with it in some capacity.
I do have some real-world experience; my home file server holds its data in a 5 drive, ~15 TB storage space. Unfortunately since I'm using it as a read-heavy file server instead of a write-heavy one, I can't really speak to the performance characteristics for your scenario.
-
@Polygeekery Fianceé's dog grooming business.
Current install is a Lorex system with 8 analog allegedly 1080p cameras, pointed at every workstation, the kennel banks, and the cash registers.
It's just about suitable for "look lady your dog got hurt because it is an asshole" or "fuck you here's a video of your monster eating my employee's hand, don't come back" and workers comp claims ("yup. Dumbass tried to bodily lift a giant dog into the tub because the dog was too dumb to handle the steps")
It's maxed out and unsuitable for actual security purposes. Following some incidents in the parking lot and homeless people colonizing the utility room, the shopping center management has offered her a juicy rent discount if she'll put up IP cameras covering the utilities (which are actually physically in her units anyway, just without internal doors and parking lot.
I was figuring on pulling the RTSP streams off the lorex into a bigger machine with loads of storage, and doing the same thing with the new IP cams. Total of like 16 streams. Aggregate and manage with iSpy or Zoneminder.
-
@Weng said in Windows Server Storage Spaces:
Current install is a Lorex system with 8 analog allegedly 1080p cameras
Those things are garbage. Absolute garbage. The entire product range is. Those systems are how we go in to this line of work. All of those systems are just unadulterated junk so I went looking for a better way to do things.
@Weng said in Windows Server Storage Spaces:
I was figuring on pulling the RTSP streams off the lorex into a bigger machine with loads of storage, and doing the same thing with the new IP cams. Total of like 16 streams. Aggregate and manage with iSpy or Zoneminder.
Give Blue Iris a look. It is what we have been using and it works well enough.
A bit of advice, dump the analog cameras. They are a total pain in the dick to work with and trying to bring those streams in to a workstation box as a DVR is more headache than it is worth. We tried several solutions, all of them were horrible.
Although, to be fair, when we tried that integration we also needed to bring in audio on those feeds, which made it much more complicated. The only cards we could find at the time came from some Asian supplier that was 12 hours offset from us and his English proficiency was somewhere around 3%. This was also several years ago. But still, scrap the analog cameras. You will spend way more money and time trying to bring those in to the new system than you would just starting over.
Reolink makes some decent cameras that you can pick up off Amazon for cheap. We have several that have been deployed for 3+ years without a hiccup. If you don't need audio they are even cheaper.
-
@Weng said in Windows Server Storage Spaces:
Zoneminder
I've actually been having troubles getting that to be stable, though I think it might be bad interaction with my cameras...
-
@Tsaukpaetra said in Windows Server Storage Spaces:
@Weng said in Windows Server Storage Spaces:
Zoneminder
I've actually been having troubles getting that to be stable, though I think it might be bad interaction with my cameras...
It is not an intuitive software to work with. At some point I should write up the whole tale of how we got in to this particular segment of IT. Zoneminder was part of the story.
-
@Tsaukpaetra said in Windows Server Storage Spaces:
@Weng said in Windows Server Storage Spaces:
Zoneminder
I've actually been having troubles getting that to be stable, though I think it might be bad interaction with my cameras...
It is equally likely that it is entirely Zoneminder's fault. You don't so much configure Zoneminder and tell it what to do. It is more that you trick it in to doing what you wish it would do, if you are lucky, and all the stars align.
-
@Polygeekery Planning on phasing out the analogs over time. The cameras have a finite lifespan in the bathing room because they aren't properly sealed against high humidity and dog dander driven by high velocity dryers. Run them in there until the inside of the lens clouds over, pitch them, rotate in another camera and replace it with an IP cam. And then when the last ones die, replace them with proper outdoor sealed cameras
The lorex actually manages to output a perfectly usable RTSP stream once you literally ignore every feature it has and figure out that it truncates passwords in the admin UI and when setting them, but not when checking credentials on RTSP. In this particular case, it truncates to SIX characters.
Notably on this model the username can be 8 characters...
-
@Weng you might be fine, but from my experience the Lorex (and other similar boxes) will randomly drop the mount of the hard drive. Since you are just scooping up the RTSP streams you should be fine, as long as losing the hard drive doesn't stop the streams.
The Reolink cameras are all IP67 rated, so they should (in theory) be fine in the high humidity environment.
Plus, they are super cheap for the quality:
-
@Polygeekery said in Windows Server Storage Spaces:
The more I think about it, the more I wonder if a Drobo wouldn't be the way to go.
Nope. The latest iSCSI Drobo only supports a maximum LUN size of 16TB. I suppose I could create multiple of those and JBOD them together at the OS but we are once again in making things more complex than they need be.
Time to test out Storage Spaces and see how it fares.
That's a real shame about the Drobos. I was already seeing ways that we could have fit them in our workflows, but the 16TB limitation is a deal breaker for what we would have used them for.
-
@Polygeekery said in Windows Server Storage Spaces:
Nope. The latest iSCSI Drobo only supports a maximum LUN size of 16TB. I suppose I could create multiple of those and JBOD them together at the OS
Nope. The Drobo uses a 4k sector size for NTFS so the 16TB is a file system limitation.
-
@Polygeekery said in Windows Server Storage Spaces:
@Polygeekery said in Windows Server Storage Spaces:
Nope. The latest iSCSI Drobo only supports a maximum LUN size of 16TB. I suppose I could create multiple of those and JBOD them together at the OS
Nope. The Drobo uses a 4k sector size for NTFS so the 16TB is a file system limitation.
Sector or block? 4k is usually the default NTFS block size.
-
@mikehurley block size. Take me by what I mean, not what I say. :)
-
I know that approximately no one cares about this, but it seems that the Drobo B1200i actually supports 64TB volumes, but you have to create a generic volume and then format it through Windows after you mount it via iSCSI. If you create a volume through the Drobo Dashboard for NTFS then 16TB is the max. B800i has a max volume size of 16TB, I think. The B1200i got a late firmware update that raised limits to 128TB total storage and 64TB volumes.
Drobo documentation is absolutely horrible. They say the same things in many different places, but when something is changed they only update parts of it.
I have a friend in corporate IT for a large enterprise. I asked him about it and he said he thought they had a ton of Drobo stuff in storage that someone else procured for purposes that it was unsuited for and then left for another job so it got mothballed. He is going to check in to it and if it is fully depreciated I will "recycle" it for them.
It is a shame that Drobo documentation is so piss poor. They make great backup targets for SMB. I think they overstepped their bounds when they tried to target enterprise and that hurt their name.
Hell, if they added AD support their NAS boxes would be great for SMB also.
-
@Polygeekery Really late, but what about Synology? Too pricy?
-
@SirTwist none of the other NAS devices will support mixed drive sizes and expand on the fly and etc the way Drobos will.
-
@Polygeekery Synology does: https://www.synology.com/en-us/support/RAID_calculator Try SHR and SHR-2
-
@SirTwist interesting. I looked in to SHR before and the documents I read gave me the impression that they couldn't do that.
-
@SirTwist said in Windows Server Storage Spaces:
@Polygeekery Synology does: https://www.synology.com/en-us/support/RAID_calculator Try SHR and SHR-2
Interesting...
-
@Polygeekery The Synology can also handle the cameras, but you need rather pricy licences past the first two cameras ($370 for an 8 camera license). This however means you don't need the server to be messing with the cameras. The only experience I have with Synology is the cheapest 2-drive unit we are using as the backup server, but it's been rock solid for years now.
-
@SirTwist said in Windows Server Storage Spaces:
The Synology can also handle the cameras, but you need rather pricy licences past the first two cameras ($370 for an 8 camera license).
Yeah. I looked at them for that exact purpose a few years ago. The licenses are retarded expensive. The pricing I have seen is more like $40/camera. But still, for a basic NVR, a few hundred bucks should be the stop-loss and then the rest of the licenses are free up to the limits of the hardware. You're already selling a $1,000-$10,000 NAS unit. The per-camera licensing add-on really rubbed me the wrong way.
Amazon.com