DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one
-
@topspin said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
multiple init systems
MULTIPLE PROCESSES WITH PID 1
HOW CAN THIS BE
-
@cvi said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
As somebody pointed out, I too think that file system notifications are handled at a higher level than the actual filesystem.
The limitation to NTFS on Windows and ext4 on linux sounds much more like they want to use the alternate data streams (NTFS)/extended attributes on ext4 for storing metadata. The latter exist in some form or another in a bunch of different filesystems, but I don't know if there's a common interface or anything.
I know you didn't spend the 3 seconds to Google it, but apparently Dropbox didn't either?
-
Dropbox claims to write a lot of code in Go, right? I just wish there was a cross-platform library for filesystem watching without polling that they could use. But one can only hope.
-
Ok, I read the article and this part jumped out to me:
For a company to support Linux, they have to consider supporting:
- Multiple file systems
- Multiple distributions
- Multiple desktops
- Multiple init systems
- Multiple kernels
How little must someone know about programming to think this?
Imagine that someone is telling you about a new program they wrote that connects to their REST API. Imagine if they tell you it's really hard to write that because you need to write specific code for every user's network card.
Imagine if someone said that changing your desktop wallpaper in Windows required special support from the programs running on your computer to not crash.
Imagine if someone comes to your company and complains about your software not running smoothly on their Raspberry PI emulating a Commodore 64 emulating a PowerPC Mac OS X computer. Would you need to support them, or would you point out that what they're doing is not a supported configuration for your software?
There are two main problems with this "Linux has too many choices" argument:
-
Your program shouldn't know or care about most of these choices available to a user.
Would it make sense for a text editor to only work on AMD graphics cards?
Would it make sense for a program to only work on computers with even numbers of processor cores?No, because the program shouldn't know or care about any of that unless it's the kernel.
In a similar vein, no program that isn't providing a filesystem or dealing with partition tables should care about filesystems or disks.
-
You don't have to support every possible configuration.
I know this may shock you, but it's okay if not every single person in the world can use your software.
You don't need to go out of your way to figure out if people should be able to run your software either - just say on your website "we only officially support this configuration" and then if anyone reports a problem on an unsupported configuration, feel free to not fix whatever problem they're having if it doesn't affect your supported configuration as well.
Steam for Linux is only officially supported on Ubuntu. Does that mean that running Steam for Linux on Fedora fails? No, it just means that if you have a problem, Valve won't be in charge of fixing it for you.
And if your software consists of detecting file changes using
inotify
and then synchronizing those file changes over HTTP, you'd be hard-pressed to find an actively-used Linux distribution where that wouldn't work exactly the same way as it does on your test machine.
-
-
@ben_lubar said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Imagine if they tell you it's really hard to write that because you need to write specific code for every user's network card.
The tricky thing about that is buying all the Big Macs so you can get the addresses off of them.
-
@ben_lubar said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
I know you didn't spend the 3 seconds to Google it
etc etc
However: what I didn't know was that ext4's extended attributes were as limited as those on ext2/ext3 (namely, the total size of all attributes names+values cannot be larger than a $entry, which can be as small as 1k). I thought that ext4 would be much more liberal (for example, btrfs has a upper limit of 64k per entry and apparently no (theoretical) upper limit on the number of entries). Initially, my thinking was that they were possibly running into some limit regarding ext attrs with ext3 or whatever.
AFAIK NTFS ADS are essentially just other file streams (or rather the other way around, the file contents are just a special ADS). They can be arbitrarily sized.
@ben_lubar said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
I just wish there was a cross-platform library for filesystem watching without polling that they could use.
Not to rain on your Go-parade, but that cross platform library is useless if the kernel doesn't export the required interfaces. Which used to be a thing like 200 years ago (the Github page even mentions kernel versions 2.6.27+ and 2.6.36+ specifically for two of the non-polling methods).
Also: it seems to fall back to polling, or at least has the option to do so.
Essentially, if you want to avoid polling, you're still stuck not supporting some older kernel versions. Your cross platform library doesn't help there.
-
@cvi said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
However: what I didn't know was that ext4's extended attributes were as limited as those on ext2/ext3 (namely, the total size of all attributes names+values cannot be larger than a $entry, which can be as small as 1k). I thought that ext4 would be much more liberal (for example, btrfs has a upper limit of 64k per entry and apparently no (theoretical) upper limit on the number of entries). Initially, my thinking was that they were possibly running into some limit regarding ext attrs with ext3 or whatever.
But the real question is why encryption of the file system would matter.
-
@Atazhaia said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Another thing I cannot understand is this whole mentality of “there’s so much effort porting applications to Linux because you have support every permutation of software combination”. No, you don’t. Just port it to the major dists to cover most of the userbase.
This is exactly the mentality that gets you a Windows version only and no Linux version at all. Windows covers 99% of the user base, why would you even bother doing a Linux version at all?
-
@ben_lubar said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
In a similar vein, no program that isn't providing a filesystem or dealing with partition tables should care about filesystems or disks.
At work, we use file locking to ensure only one copy of various daemons runs. File locking is not reliable over NFS. So we tell customers they can't have that directory on NFS, and if they do anyway and file bugs we point them back at the setup instructions.
I'm not gonna test for cases like that cause I'll be at it forever.
-
@remi Yeah. The reality is if you want to make Linux popular, you have to make Linux development SO EASY that it's literally a no brainer to port to it. Ideal scenario: you make it so Linux can run any Windows binary/driver.
If Windows has a 99% market share, and Linux has a 1% market share, Linux development needs to be two orders of magnitude easier than Windows development to make it rational to port to it. Right now, it's harder than Windows development in a ton of ways that makes ports basically impossible. (Expect in situations where the engine software does it for you-- say Unreal or Unity video games.)
I mean these motherfuckers are using shit like VIM or Emacs and half of them literally don't even know what an interactive debugger is. And they're by and large writing in non-memory-managed languages like C or C++. (Although it's good that Go's making some progress, but even Go is maybe 1/3rd as good as C#.) I do development on Windows, and I cringe whenever I have to make the change from a Windows development tool like Visual Studio to a Linux one like fucking Git-- the different in ease-of-use and quality is night and day.
But when the situation is "writing the Linux version is far, far harder oh and there's only 1% of the possible benefit" well then fucking of course nobody writes software for Linux, why the shit would they? People aren't idiots, they're rational.
-
@blakeyrat said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
People aren't idiots, they're rational.
E_NO_REPRO
-
@blakeyrat said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
writing the Linux version is far, far harder
Please tell us how much experience you have writing programs for Linux
-
@TimeBandit said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Please tell us how much experience you have writing programs for Linux
I have all of the experience.
-
@blakeyrat said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
I have all of the experience.
Liar
-
@remi said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Windows covers 99% of the user base,
83%
-
@blakeyrat inotify has been there since a long ago. So are the extended attributes, moreover — XFS doesn't have many of the limitations on their size that ext4 has. That's why the whole excuse smells of red herring to me.
Dropbox keeps growing, they can afford better programmers, and at the same time they stop supporting stuff that was working perfectly well — this also is very fishy.
-
@wft said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Dropbox keeps growing, they can afford better programmers, and at the same time they stop supporting stuff that was working perfectly well — this also is very fishy.
OH NOES LINUX CONSPIRACY!!!!! DropBox is PAID CRISIS ACTORS! Linux filesystems CAN'T MELT STEEL! Filesystem monitoring events CAUSE CHEMTRAILS!!!!! VERY FISHY!
-
@blakeyrat oh, fuck off. Just fuck off.
-
I understand some of the concerns in the article. I think some are overblown, but dealing with the combinatorial logic of distribution, kernel versions, package manager, and such takes so much time and is so error-prone that often I question whether supporting Linux is actually worth our effort. And many Linux users endlessly tinker with stuff just for the sake of tinkering, and those ones really drive up support costs because they get their systems into unreasonable configurations that should never be allowed in the first place, but they're paying customers so we have to help.
(We do kernel-mode device drivers and hardware APIs, not general-use user-mode applications.)
-
@wft said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
@blakeyrat oh, fuck off. Just fuck off.
Did the Linux Illuminadi get to you!? They replaced the queen with a lizard-woman and build tunnels between Hong Kong and Washington DC to hide stolen Nazi gold from the Rothschilds!!!
-
@sockpuppet7 But the actual relevant figure is Linux 1-2% (if you're generous you could say that Linux is actually 100-Windows-Mac, so more like 4-5%).
-
@remi How much work was it to officially drop support for most filesystems and detect when you're on a filesystem you don't support?
How much work would it have been to just not support that stuff?
-
@ben_lubar said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
How much work would it have been to just not support that stuff?
That would damage their brand when people see dropbox break in an unsupported scenario.
-
@sockpuppet7 said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
@ben_lubar said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
How much work would it have been to just not support that stuff?
That would damage their brand when people see dropbox break in an unsupported scenario.
Yes. They could have left a patchy support for some FS, at an immediate cost of 0 (no work, just leave code), but at a significant long term cost both in actual support ("oh, yeah, after 10 emails I can tell you why you have issues, you are not using the correct FS") and in reputation ("42 bazillions results for your query of 'dropbox linux broken'", all pointing to looooong threads either without an answer or with someone finding out about the FS thing after 100 posts).
Or they could have spent a couple of hours finding how to detect properly the FS (and I'm being generous with that... if their code already depends on the FS, they must necessarily already have the FS-detection code in place), and redirect to a single FAQ item ("we don't support this, fuck off"). In both cases they lose all the non-ext4 users, but again I assume they know how much that really is (probably peanuts), but by clearly removing support they insulate themselves from all sorts of later headaches.
It all comes back to whether they really do FS-specific stuff. If they don't, then restricting themselves to a single FS is a huge . If they do, and it seems from this thread that they would have acceptable reasons to do so, then it is both very understandable, and very straightforward, to limit themselves to a single one.
-
@blakeyrat said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
@wft said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Dropbox keeps growing, they can afford better programmers, and at the same time they stop supporting stuff that was working perfectly well — this also is very fishy.
OH NOES LINUX CONSPIRACY!!!!! DropBox is PAID CRISIS ACTORS! Linux filesystems CAN'T MELT STEEL! Filesystem monitoring events CAUSE CHEMTRAILS!!!!! VERY FISHY!
-
@remi I was using Dropbox for YEARS on XFS. No problems whatsoever. Now out of the blue they say that I’m delusional and it’s in fact broken. In my opinion they are fixing what’s not broken in the first place.
OK, I can just say fuck Dropbox. It’s just glorified rsync for me, so I can just use, well, rsync.
-
@TimeBandit said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Finally, it never ceases to amaze me that Linux has yet to take hold on the desktop market.
I've seen less than a handful of linux desktop/laptops on a pc retailer shelf in my entire life. This is why.
-
@xaade said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
I've seen less than a handful of linux desktop/laptops on a pc retailer shelf in my entire life. This is why.
Exactly.
But you can order them from online retailers like System76, or even Dell (if you REALLY search for it)
-
@blakeyrat said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Also, why would I need a checked build of the OS for normal application development?
... why wouldn't they?
But the point I was making is: Linux has nothing equivalent to that. Linux tooling sucks.
You mean options like debugfs, kernel debug symbols, lockup detection, scheduler statistics, memory leak debugging, mutex and spinlock debugging all don't exist? Funny, they were there in my kernel config menu last time I looked.
Now if Dropbox was big enough to convince MS to let them have the usual bureaucratic, heavily NDA'd peek at the NT kernel source …
-
@LaoC said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
@blakeyrat said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Also, why would I need a checked build of the OS for normal application development?
... why wouldn't they?
But the point I was making is: Linux has nothing equivalent to that. Linux tooling sucks.
You mean options like debugfs, kernel debug symbols, lockup detection, scheduler statistics, memory leak debugging, mutex and spinlock debugging all don't exist? Funny, they were there in my kernel config menu last time I looked.
Now if Dropbox was big enough to convince MS to let them have the usual breaucratic, heavily NDA'd peek at the NT kernel source …
apt-add-repository http://ddebs.ubuntu.com
Now let's see, what's the command to enable Windows debug symbols in the OS's package manager?
-
@wft said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
@remi I was using Dropbox for YEARS on XFS. No problems whatsoever. Now out of the blue they say that I’m delusional and it’s in fact broken. In my opinion they are fixing what’s not broken in the first place.
Yes, exactly like Microsoft suddenly saying that my Windows 7 is broken and that I must update to Windows 10, or Android saying that the lock screen notifications that worked fine before were in fact broken and removing them in the next version.
It sucks. The process by which software gets changed without the user having any say in it, without developers giving even a tiny fuck about users, that whole system and process sucks. Hard. And hardly anyone seems to complain about it, which IMO sucks even more.
Is Dropbox more at fault and sucky-ier than the rest? Not that I can see.
OK, I can just say fuck Dropbox. It’s just glorified rsync for me, so I can just use, well, rsync.
If you're not using all features of Dropbox, that might be why you never saw it being broken on your FS in the first place. Maybe if you used it as more than that you'd have seen cases where your XFS causes issues (slowness, disk churn, whatever).
-
@ben_lubar said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Now let's see, what's the command to enable Windows debug symbols in the OS's package manager?
-
@remi said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
@wft said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
OK, I can just say fuck Dropbox. It’s just glorified rsync for me, so I can just use, well, rsync.
If you're not using all features of Dropbox, that might be why you never saw it being broken on your FS in the first place. Maybe if you used it as more than that you'd have seen cases where your XFS causes issues (slowness, disk churn, whatever).
What are the other features of dropbox? Serious question, because my understanding is that it's basically automated and in the background rsync (plus online availability).
-
@boomzilla Given the number of updates I see (on my tablet), with messages about how they added shiny new ideas, I guess there must be other features.
But tbh, while I do have a dropbox account, I don't use it at all (except maybe twice a year if someone sends me something through it), so I don't have the slightest clue as to what those other features are.
(following the usual laws of software development, they probably include an
email clientsocial network of some form...)
-
@remi Hmmm...
File and version recovery
Easily recover deleted files and restore previous file versions.File versioning looks like the only fancy pants thing on there that might require some filesystem level shenanigans to manage locally.
-
@TimeBandit said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
@blakeyrat said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
writing the Linux version is far, far harder
Please tell us how much experience you have writing programs for Linux
Zero. Hence it's far far harder. In my very non-technical user base, I'd guess I have 1 - Ok, maybe 2, users who even know what Linux is (and they have either a Mac or Window anyways - or know how to run programs with Wine). So I have zero motivation to learn how to install/configure/use Linux (let alone figure out which of the 25kajillion varieties I should use).
-
@boomzilla said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
@remi Hmmm...
File and version recovery
Easily recover deleted files and restore previous file versions.File versioning looks like the only fancy pants thing on there that might require some filesystem level shenanigans to manage locally.
File versioning? You mean like what Windows claims it can do but backs up 0 bytes of data when I tell it to start? Or what BTRFS supports by default with no overhead?
-
@remi said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Maybe if you used it as more than that you'd have seen cases where your XFS causes issues (slowness, disk churn, whatever).
Oh the irony. I tried to keep a git repository on it once. Mind you, it was on ext4 on my Linux machines, so reportedly that one filesystem that was supposed to be without a glitch... And it fucked it up big time. No amount of
git fsck
could make it right. I knew better since and never again kept my.git
directories on it.I knew better since, so it became my rsynced folder for "backup" stuff — like scans of paper documents I should keep just in case. I'm now considering to move those over to my OneDrive and use with its web interface exclusively. (And there's a whopping terabyte available!)
-
@boomzilla there was one, but I only used it on a Mac. Dunno if it could work on Linux at all. You make a screenshot, and it could upload and share it instantly. A godsend when reproducing bugs or proving to the customer that this one feature he wants implemented was in fact a thing for the last 6 months.
-
@wft said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
@remi said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
Maybe if you used it as more than that you'd have seen cases where your XFS causes issues (slowness, disk churn, whatever).
Oh the irony. I tried to keep a git repository on it once. Mind you, it was on ext4 on my Linux machines, so reportedly that one filesystem that was supposed to be without a glitch... And it fucked it up big time. No amount of
git fsck
could make it right. I knew better since and never again kept my.git
directories on it.I knew better since, so it became my rsynced folder for "backup" stuff — like scans of paper documents I should keep just in case. I'm now considering to move those over to my OneDrive and use with its web interface exclusively. (And there's a whopping terabyte available!)
Don't put anything that journals in Dropbox. No database servers, no Git repositories, no browser profile folders.
Copying stuff there as a backup is fine, but the same rule applies: don't allow anything that might modify the files while you're copying them to run.
Journaling will save you from crashes, but only if the operating system's sync() function doesn't lie, which is what copying files while they're being modified does.
-
@boomzilla said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
What are the other features of dropbox? Serious question, because my understanding is that it's basically automated and in the background rsync (plus online availability).
It's much like the other online personal/business file sharing services:
- It syncs a set of files between multiple machines.
- You can choose to only sync a subset of your files on a per-device basis.
- It keeps track of multiple versions of these files, allowing you to restore back to any saved version. Versions aren't saved forever, though.
- Similarly, it lets you restore deleted files.
- You can have it back up your pictures and videos when appropriate devices are connected (desktop) or from the device itself (mobile).
- It lets you share files and folders with others via links, allowing them to download (or copy to their Dropbox account) whatever you've shared.
- It also lets you share with restrictions to other Dropbox users. With appropriate permissions they can comment on or edit files as well as make copies.
- They have their own Office equivalent in Dropbox Paper, letting you work with some types of documents online (including collaboration with other users).
- You can allow it to show notifications for when any of this stuff happens.
Much of this is done online, though. The current Windows client only handles a few of these, sending you to the website for the rest. What I don't know is how much of the file handling is done locally vs. online.
-
@cvi said in DropBox dropping (GET IT? GET IT? GET IT?) all Linux filesystems and file encryption except one:
And, just to clarify, dropbox using them is just speculation. I don't know if that's the reason - I don't have the dropbox client installed or anything. If somebody does, it should be easy to verify. Just check if the files managed by the dropbox client have non-standard ADSs or extended attributes on them.
It looks highly likely to me; on OSX, Dropbox adds a
com.dropbox.attributes
extended attribute to each file and directory with 83 bytes of data in it. The contents look pretty random to me, so I guess a lot of that is some sort of crypto hash of the content so that the Dropbox client can figure out definitively whether the file or directory has been meaningfully modified. The rest might be some sort of ID used for mapping things on the server side? Or some control flags? Hard to say and I'm not interested in poking deeper. ;)In any case, since they use that strategy on one platform, it's going to be unsurprising if they do something similar on others.
what happens if you copy the file to a filesystem that doesn't support them, or pass the file through some application that doesn't know about those
Then they get stripped. Which is fine; the Dropbox client then knows to recompute from the content and potentially by communicating with the service. In fact, this (cached extended content-derived metadata) seems to me to be a perfect use case for these kinds of things; using ordinary auxiliary files would be a lot more likely to go nastily wrong…