Getting frustrated with network filesystems
-
Dear forums,
I'm in the market for a network filesystem. The server is Linux, the clients are Linux and macOS, sometimes simultaneously. What I basically need is:
- Does not suck camel balls when I have lots and lots of small files to throw around (which rules out most FUSE stuff like SSHFS and Apple's AFP)
- Does not make your server a hole full of security holes (which rules out NFS)
- Does not suck balls when you use it over the internets (which rules out almost everything there is)
- Allows specifying explicitly on which addresses to listen (which rules out NFS again)
- Does not suck balls when mmap() or sendfile() is used on it (which probably only leaves CIFS)
- Supports all kinds of links, POSIX ACLs, and extended attributes (CIFS probably is the only one that does this)
- Does not screw up Git repositories (the directories I need to share are working copies)
- Does not take weeks to tune properly (which leaves out all the fancy "cluster filesystems")
The context is: I log on to an IPSec VPN which I have, connect to the file server, and it happily and reliable allows me to work on my projects with any of the five laptops I posess and use, and other servers immediately show me the results (they have their own means for syncing up which are not quite networked and thus are out of scope here). I have nice bandwidth most of the time, but its in the internets, and the internets like to introduce latencies.
Also, what I need is: I change a file on my laptop, save it, and inotify on the server side says a file has changed. Reliably. Every single time.
While I'm almost sure I need to go with CIFS, I'd like to ask for experiences and impressions. Have you ever used any networking FS over the Internet (VPN or otherwise)? How did it feel?
-
CIFS blows goats over any link with latency
-
rsync user@remote:/remote/project/path /local/project/path watch rsync /local/project/path user@remote:/remote/project/pathYou can thank me later.
-
@wft said in Getting frustrated with network filesystems:
I'm in the market for a network filesystem.
They all suck dead donkey balls. You can do a lot of research and everything, but ultimately the problem is that networks both can and do go horribly wrong in far more ways than local disks ever can.
-
If you only need single user, iSCSI apparently works astonishingly well.
In principle, you can do multiuser iSCSI with the appropriate (local) filesystem I've never done it myself.
-
Yes, when you discover the answer to this pipe dream please let me know!
-
@weng said in Getting frustrated with network filesystems:
CIFS blows goats over any link with latency
CIFS, the 1996 protocol designed for LANs? Yes, it probably does. Try SMB instead.
Why You Should Never Again Utter The Word, "CIFS" - Stephen Foskett, Pack Rat
CIFS is not the network storage protocol used by Microsoft Windows, and many other clients. The protocol used to share files over a LAN by the majority of personal computers is called SMB. I wish everyone in the industry would get that through their heads.
-
@anonymous234 I am aware, yes.
Our ITSec team isn't, though.
SMB also blew goats over high latency links last I tried.
Which, admittedly, weren't SMBv3.
-
@weng said in Getting frustrated with network filesystems:
Our ITSec team isn't, though.
Stab them in the face.
-
@anonymous234 I somehow failed to give a fuck.
-
Ok, I'll ask the dumb question: why not just clone to each of your computers? Or, assuming you have access, why not just have one computer (the beefiest probably) be the workstation and VNC or RDP into it from the others? Seems odd to have that many computers that you could be using to develop whatever you're developing at nearly the same time.
-
@weng said in Getting frustrated with network filesystems:
SMB also blew goats over high latency links last I tried.
It works well over low-latency high-bandwidth links, at least until something craps out in the network config and then you're still screwed.
-
@anonymous234 They also want me to provide port numbers for it every fucking time it comes up.
Like, fuckers it's a standard fucking protocol.
-
@wft I've been in your situation before, and SSHFS on FUSE really is the best option I've found. I use SSHFS for remote file editing almost every day (but it's usually just over the LAN).
Git is another option.
-
@captain said in Getting frustrated with network filesystems:
SSHFS on FUSE really is the best option I've found.
I've had problems with that in the past. Eventually worked around them by configuring things so that the SSH connection used aggressive monitoring to detect issues, would be dropped on the slightest glitch, and the FUSE config would then relaunch the connection immediately. That turned the flakiest network filesystem in that deployment into the most reliable one.
Also, I did some tricky magic to stop a particular network-level config fuck-up from causing problems, but I don't expect those to be a normal problem for anyone. After all, nobody in their right minds would put a pool of SSH servers behind a stupid mandatory DNS round-robin, would they?
-
@dkf said in Getting frustrated with network filesystems:
I've had problems with that in the past. Eventually worked around them by configuring things so that the SSH connection used aggressive monitoring to detect issues, would be dropped on the slightest glitch, and the FUSE config would then relaunch the connection immediately. That turned the flakiest network filesystem in that deployment into the most reliable one.
There are a few SSH options that are helpful too. I hated when the underlying SSH connection timed out and I had to wait like 5-15 seconds for SSH to figure out I was asking it to do something.
The biggest problem with SSHFS is that all traffic is encrypted, so it's pretty slow (in KB/s more than latency). This is also a strength, though.
I do not remember the details. I used Macfusion to control/mount the SSHFS disk on my old mac. But that program doesn't really exist anymore. So for now, I just do
sshfs myXeonDevServer .
-
@captain said in Getting frustrated with network filesystems:
it's pretty slow
I don't remember that as being a problem. The overhead's quite a bit when the connection is being established, but after that is acceptable IME.
-
@captain said in Getting frustrated with network filesystems:
The biggest problem with SSHFS is that all traffic is encrypted, so it's pretty slow (in KB/s more than latency).
? If must be encrypted to shit and back if this is even measurable.
I mean, everything the Xbox sends is encrypted and people are playing Call of Duty on that beast.
-
@dkf said in Getting frustrated with network filesystems:
@captain said in Getting frustrated with network filesystems:
it's pretty slow
I don't remember that as being a problem. The overhead's quite a bit when the connection is being established, but after that is acceptable IME.
I've ran into SSH encryption overhead as a bottleneck, but that was when I was trying to fill up a gigabit line toward an atom-powered nas without hardware encryption support.
-
@cartman82 Basically using Unison now, which rocks.
-
@wft said in Getting frustrated with network filesystems:
@cartman82 Basically using Unison now, which rocks.
Immediate crippling problem with unison:
(on a random server)
-
@cartman82 said in Getting frustrated with network filesystems:
@wft said in Getting frustrated with network filesystems:
@cartman82 Basically using Unison now, which rocks.
Immediate crippling problem with unison:
(on a random server)apt-get unisom??? :)
-
@tsaukpaetra said in Getting frustrated with network filesystems:
apt-get unisom??? :)merlin@emrys:~$ sudo apt-get install unisom Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package unisom
-
@heterodox You need to have the de-accalia package installed for that sort of thing to wrok.
-
@boomzilla said in Getting frustrated with network filesystems:
@heterodox You need to have the de-accalia package installed for that sort of thing to wrok.
I guess there's always:
merlin@emrys:~$ unisom-2.40 No command 'unisom-2.40' found, did you mean: Command 'unison-2.40' from package 'unison' (main)(Had to cheat, I guess the unison package doesn't actually install an executable named unison or it's a symlink so
command-not-foundwon't find it.)
-
@blakeyrat said in Getting frustrated with network filesystems:
@captain said in Getting frustrated with network filesystems:
The biggest problem with SSHFS is that all traffic is encrypted, so it's pretty slow (in KB/s more than latency).
? If must be encrypted to shit and back if this is even measurable.
I mean, everything the Xbox sends is encrypted and people are playing Call of Duty on that beast.
I mean maybe whatever implementation they're using creates a new TCP connection for each file operation?
-
@boomzilla said in Getting frustrated with network filesystems:
@heterodox You need to have the de-accalia package installed for that sort of thing to wrok.
downloafanduploafare actually in the upack spec for verbs that the server has to understand.
-
Urban Dictionary: Downloaf