Microsoft Outlook and APFS
-
Inside macOS 10.13 High Sierra: APFS benefits end users with space, speed | AppleInsider
Apple's next-generation APFS has made its way to macOS High Sierra after an official debut on iOS 10.3, and with it comes essentially instant file copies, better efficiency for greater overall speed, and fine-tuning of read and write operations boosting system performance.
There are some other sundry software bugs associated with APFS. Third-party utilities that hit the file system hard need to be updated with Carbon Copy Cloner already supporting the file system in its own beta. Microsoft Outlook won't load message bodies from an APFS volume —but that's up to Microsoft to fix.
Which has me wondering: what could possibly be the reason for Outlook’s ability to read messages to depend on the file system you use …?!
-
-
@gurth said in Microsoft Outlook and APFS:
Which has me wondering: what could possibly be the reason for Outlook’s ability to read messages to depend on the file system you use …?!
Maybe they used some concept like alternate streams on the previous filesystem whereas this new filesystem uses them in another way? It's all speculation, but I think it's not beyond Apple to break backwards compatibility.
-
@jbert said in Microsoft Outlook and APFS:
@gurth said in Microsoft Outlook and APFS:
Which has me wondering: what could possibly be the reason for Outlook’s ability to read messages to depend on the file system you use …?!
Maybe they used some concept like alternate streams on the previous filesystem whereas this new filesystem uses them in another way? It's all speculation, but I think it's not beyond Apple to break backwards compatibility.
I'd say it's rather SOP for Apple to break backwards compatibility.
-
@jbert said in Microsoft Outlook and APFS:
It's all speculation, but I think it's not beyond Apple to break backwards compatibility.
True, but that’s not what’s got me puzzled. Why would an email client need to access the disk in such a way that the file system in use matters?
-
@gurth said in Microsoft Outlook and APFS:
Why would an email client need to access the disk in such a way that the file system in use matters?
It doesn't need to.
It doesn't in practice.
And Apple broke it somehow.
Outlook does use a "real" (more or less) database to store data, unlike most email clients that just use a huge .mbox text file, so they've probably broken some filesystem assumption that Outlook's database relies on.
But here's the deal: what about actual databases? Do they all work? Did Apple employees do a flash-effort to patch them all in the last few weeks to ensure they'd keep working? There's nothing Outlook's doing that MySQL or Postgres or MongoDB isn't also doing.
So yes it's very worrying, and if I were more inclined to paranoia I'd almost think it was sabotage on Apple's part.
-
Comments on article suggest it is a change from case insensitive to case sensitive.
-
@gurth said in Microsoft Outlook and APFS:
@jbert said in Microsoft Outlook and APFS:
It's all speculation, but I think it's not beyond Apple to break backwards compatibility.
True, but that’s not what’s got me puzzled. Why would an email client need to access the disk in such a way that the file system in use matters?
Mailboxes (.pst files) are essentially the largest database common users ever create/access. We've previously discussed mailboxes being up to 30 GB. That's a shitload of text, and presumably needs special file access optimization.
I'd honestly expect nothing less.
-
@greybeard said in Microsoft Outlook and APFS:
Comments on article suggest it is a change from case insensitive to case sensitive.
Those commenters are full of shit. I mean changing to case-sensitive is absolutely moronic, sure, but how could that possibly affect a binary database file? There's only one file name relevant here.
-
@blakeyrat It could if the program was moronically changing the case of the filename before opening it.
-
@blakeyrat Maybe Outlook keeps the metadata in one file and the content in a different one?
-
@anonymous234 I suppose it's possible the macOS version does something demented that the Windows version doesn't.
-
@blakeyrat like assuming the underlying operating system had case insensitive behaviour which it now definitely doesn't...
-
@blakeyrat said in Microsoft Outlook and APFS:
Those commenters are full of shit. I mean changing to case-sensitive is absolutely moronic, sure, but how could that possibly affect a binary database file? There's only one file name relevant here.
-
@arantor said in Microsoft Outlook and APFS:
@blakeyrat like assuming the underlying operating system had case insensitive behaviour which it now definitely doesn't...
I mean, unless your software is meaninglessly altering the case of the recorded file name it uses between creation and access, why would this even matter?
If your software goes "OS, Create me a file called mydata.wack" and then later on comes along and says, "OS, Open me the file called MyData.Wack", the program is in the wrong, not the OS.
-
@blakeyrat likely because parts of the app try to load from the database using the wrong filename and it falls over where it wouldn't on a case insensitive system.
-
@tsaukpaetra said in Microsoft Outlook and APFS:
If your software goes "OS, Create me a file called mydata.wack" and then later on comes along and says, "OS, Open me the file called MyData.Wack", the program is in the wrong, not the OS.
Nope, still the OS. (Or, more specifically the filesystem.) Case-sensitive file systems are moronic idiot open source-y "fuck our users" thinking. Which is why it wouldn't surprise me one bit if Apple made that the default.
-
@arantor said in Microsoft Outlook and APFS:
using the wrong filename
Case. In. Point.
@blakeyrat said in Microsoft Outlook and APFS:
Case-sensitive file systems are moronic idiot open source-y "fuck our users" thinking.
I suppose it should also allow referencing a file named
rechtsberatung_zürich.phpasrechtsberatung_zürich.phptoo?Wait a second, this conversation seems oddly familiar...
-
@tsaukpaetra that reminds me, someone around here said that macOS stored in UTF-8 but it used the denormalised form rather than the normalised form.
This would certainly qualify too.
-
@tsaukpaetra Yeah, we've had this thread before. Case insensitive filesystems are TRWTF because case insensitivity rules differ by language.
-
Re: Filesystem case debate.
As a user of a proper GUI, why should I care if the filesystem is case-sensitive or case-insensitive-yet-preserving?
if I'm just browsing or clickering, the case surely doesn't matter.
If I'm copying the path and then pasting it in the "address bar" of another window, the case still doesn't matter.
If I'm trying to type something up in the "address bar" of an explorer window, a proper UI will show suggestions if what I'm typing up isn't quite right - such suggestions will include the correct case if needed, and the UI can even auto-correct me if it wishes to. Case barely matters (just one of the many mistakes I will make) and doesn't matter at all if the UI auto-corrects me.
-
@arantor said in Microsoft Outlook and APFS:
@tsaukpaetra that reminds me, someone around here said that macOS stored in UTF-8 but it used the denormalised form rather than the normalised form.
This would certainly qualify too.
Why the hell would it matter?
Shirley every filesystem API therefore decomposes it's inputs.
-
@blakeyrat said in Microsoft Outlook and APFS:
There's nothing Outlook's doing that MySQL or Postgres or MongoDB isn't also doing
How do you know this?
-
@boomzilla I've used all three programs.
-
@blakeyrat and now you are familiar with their implementation details? Wow, talk about a power user!
-
@boomzilla Did I say I was?
I said they all do the same thing. Not "they are all implemented the exact same way". But why not just make-up something stupid, pretend I said it, and type a few more replies. I mean why not.
-
@blakeyrat said in Microsoft Outlook and APFS:
@boomzilla Did I say I was?
I said they all do the same thing. Not "they are all implemented the exact same way". But why not just make-up something stupid, pretend I said it, and type a few more replies. I mean why not.
Don't be retarded. You implied exactly what I said. If you don't want people to mock the stupid things you say then stop saying stupid things.
-
@boomzilla said in Microsoft Outlook and APFS:
@blakeyrat said in Microsoft Outlook and APFS:
There's nothing Outlook's doing that MySQL or Postgres or MongoDB isn't also doing
How do you know this?
Well, I mean, obviously Outlook accesses files. MySQL and Postgres and MongoDB accesses files, clearly all relevant information about what they're doing is right there on the card. Exactly the same thing they're doing.
-
@weng said in Microsoft Outlook and APFS:
@arantor said in Microsoft Outlook and APFS:
@tsaukpaetra that reminds me, someone around here said that macOS stored in UTF-8 but it used the denormalised form rather than the normalised form.
This would certainly qualify too.
Why the hell would it matter?
Shirley every filesystem API therefore decomposes it's inputs.
Actually I'd assume anything that used UTF-anything would use normalised form of characters rather than renormalised forms. But I digress.
What if different parts of the API for APFS handled WTF-16 input differently?
-
@blakeyrat said in Microsoft Outlook and APFS:
Nope, still the OS. (Or, more specifically the filesystem.) Case-sensitive file systems are moronic idiot open source-y "fuck our users" thinking. Which is why it wouldn't surprise me one bit if Apple made that the default.
The screenshot on the article seems to suggest that case insesitive is the default, with an option to make it case sensitive
-
@jaloopa said in Microsoft Outlook and APFS:
The screenshot on the article seems to suggest that case insesitive is the default, with an option to make it case sensitive
I haven’t looked into it, but it’s what I would expect given that HFS+ — sorry, I mean Mac OS Extended is as well. It’s long been known there are OS X applications that don’t work correctly (either partially or not at all) if the file system is case-sensitive. Opera didn’t a decade ago, for example, but I have no idea if that’s still so. Apple making their new FS case-sensitive by default would not be the best move to encourage people to adopt it.
-
@weng said in Microsoft Outlook and APFS:
@arantor said in Microsoft Outlook and APFS:
@tsaukpaetra that reminds me, someone around here said that macOS stored in UTF-8 but it used the denormalised form rather than the normalised form.
This would certainly qualify too.
Why the hell would it matter?
Shirley every filesystem API therefore decomposes it's inputs.
Even if they do, there's a risk if the application lists a directory and evaluates the contents, since that will yield the modified values.
-
@boomzilla said in Microsoft Outlook and APFS:
You implied exactly what I said.
I did not, and I am not responsible for the contents of your brain.
-
@blakeyrat said in Microsoft Outlook and APFS:
@boomzilla said in Microsoft Outlook and APFS:
You implied exactly what I said.
I did not, and I am not responsible for the contents of your brain.
I never said that you were. I'm just reading your posts, but I guess I forgot that you don't communicate like a hooman.
-
@jaloopa said in Microsoft Outlook and APFS:
The screenshot on the article seems to suggest that case insesitive is the default, with an option to make it case sensitive
You realize you're telling this to a person who just said he thinks the comments blaming this bug on case-sensitivity were full of shit, right? Then quoted himself saying that again, since the message didn't sink in the first time.
Apparently it didn't sink in the second time either. Fun.
-
@blakeyrat yeah, but you also said (in the bit I quoted you saying) that you wouldn't be surprised if Apple made it case sensitive by default. I was providing additional context to show they're not doing that. It's kind of tangential to whether case sensitivity is the cause of the issue
-
@pleegwat said in Microsoft Outlook and APFS:
@weng said in Microsoft Outlook and APFS:
@arantor said in Microsoft Outlook and APFS:
@tsaukpaetra that reminds me, someone around here said that macOS stored in UTF-8 but it used the denormalised form rather than the normalised form.
This would certainly qualify too.
Why the hell would it matter?
Shirley every filesystem API therefore decomposes it's inputs.
Even if they do, there's a risk if the application lists a directory and evaluates the contents, since that will yield the modified values.
Howso? The listing will ultimately be used for one of two things:
- To present as an output, where it will look exactly the same to any user.
- To pass back into a filesystem API that will renormalize it to the decomposed state.
-
I read the title as "Microsoft Outlook and APDB".
-
@weng If the file names are determined by the user, there's no problem. If they are determined by an automated process (EG files picked up from FTP, with some metadata recorded in the file name) this can cause problems.
-
@weng said in Microsoft Outlook and APFS:
@pleegwat said in Microsoft Outlook and APFS:
@weng said in Microsoft Outlook and APFS:
@arantor said in Microsoft Outlook and APFS:
@tsaukpaetra that reminds me, someone around here said that macOS stored in UTF-8 but it used the denormalised form rather than the normalised form.
This would certainly qualify too.
Why the hell would it matter?
Shirley every filesystem API therefore decomposes it's inputs.
Even if they do, there's a risk if the application lists a directory and evaluates the contents, since that will yield the modified values.
Howso? The listing will ultimately be used for one of two things:
- To present as an output, where it will look exactly the same to any user.
- To pass back into a filesystem API that will renormalize it to the decomposed state.
What if a program is using Unicode code points as a way of storing metadata about a file?
-
@pleegwat said in Microsoft Outlook and APFS:
@weng If the file names are determined by the user, there's no problem. If they are determined by an automated process (EG files picked up from FTP, with some metadata recorded in the file name) this can cause problems.
Elaborate on those problems because I'm not seeing them.
-
@ben_lubar As in "I'm going to have ä in the filename contextually mean something different than a and a combining umlaut"? Then it's fucking insane. Because the meaty human who has to debug that shit can't see the difference.
Any developer who thinks that's a good idea needs to be killed for the betterment of the species.
-
@weng said in Microsoft Outlook and APFS:
@pleegwat said in Microsoft Outlook and APFS:
@weng If the file names are determined by the user, there's no problem. If they are determined by an automated process (EG files picked up from FTP, with some metadata recorded in the file name) this can cause problems.
Elaborate on those problems because I'm not seeing them.
It's not immediately obvious, which is why Apple probably thought they could get away with it. Also I definitely recommend against using non-ascii filenames for generated files if you can avoid it.
The problem arises where you are scanning for incoming file names, and matching them against a list of expected filenames.
It also arises (and this has been featured here somewhere) if a mac has been an intermediate station (so the mac denomalized the file name, then it's copied on to a windows box, and windows is asked foräwhich it won't match witha¨
-
@weng said in Microsoft Outlook and APFS:
@ben_lubar As in "I'm going to have ä in the filename contextually mean something different than a and a combining umlaut"? Then it's fucking insane. Because the meaty human who has to debug that shit can't see the difference.
Any developer who thinks that's a good idea needs to be killed for the betterment of the species.
I decided to try this on macOS 10.12: I made a file
ä.txtwhose name contains LATIN SMALL LETTER A WITH DIAERESIS and then tried saving a fileä.txtwith LATIN SMALL LETTER A plus COMBINING DIAERESIS. For the second one, it asks whether I want to overwrite the existing file.In fact, even in Text Editor (and, I suppose all other Cocoa applications) it turns LATIN SMALL LETTER A plus COMBINING DIAERESIS into a single
ächaracter that gets deleted in its entirety with one press of the Backspace key.
-
@weng said in Microsoft Outlook and APFS:
@ben_lubar As in "I'm going to have ä in the filename contextually mean something different than a and a combining umlaut"? Then it's fucking insane. Because the meaty human who has to debug that shit can't see the difference.
Any developer who thinks that's a good idea needs to be killed for the betterment of the species.
The BARF compression algorithm can compress any file to a smaller size by cheating on what file size is defined by.
-
@pleegwat said in Microsoft Outlook and APFS:
The problem arises where you are scanning for incoming file names, and matching them against a list of expected filenames
Breaking news - conform to the standards of the platform on which you are running.
Your "beautiful" "portable" code ain't.
-
@weng Meh, if you're going to parse/match your incoming filenames just stick to ASCII, without controls, whitespace, uppercase, and everything Windows disallows. I imagine this kind of stuff mostly arises when people either don't realize what they're putting in their filenames, or are subject to technical requirements from PM.
-
@pleegwat filenames for anything other than dates, types and basic categorization is completely and utterly stupid.
-
@weng We end up putting a load of stuff into it because a lot of generated/derived configuration gets materialized in files. But all human-entered data gets hashed before it goes into a filename; only development-sourced strings and integers go direct.
-
@arantor said in Microsoft Outlook and APFS:
denormalised form
It uses UTF-8, Normalised Form Decomposed (NFD). It means that it stores diacriticals and other related marks separately from the base character to which they are applied.
No way would this be a sane reason for breaking an app, especially if the app is being run by people in the US who stick to the printable ASCII subset like heathens.
