Linux/Unix/POSIX files



  • I just came across this awesome essay explaining just how difficult dealing with POSIX filenames is. I'm not even all the way through, and the WTF quantity is very high. Some choice quotes:

    @DWheeler said:

    The list doesn’t include “hidden” files (filenames beginning with “.”), but often that’s what you want anyway, so that’s not unreasonable. The problem with this approach is that although this usually works, filenames could begin with “-” (e.g., “-n”). So if there’s a file named “-n”, and you’re using GNU cat, all of a sudden your output will be numbered! Oops; that means on every command we have to disable option processing. The “obvious” way to do this is to litter command invocations with “--” before the filename(s). But it turns out this doesn’t really work, because not all commands support “--” (ugh!). For example, the widely-used “echo” command is not required to support “--”. What’s worse, echo does support at least one dash option, so we need to escape leading-dash values somehow. POSIX recommends that you use printf(1) instead of echo(1), but some old systems do not include printf(1). In my opinion, a much better solution is to prefix globs like this with “./”. In other words, you should do this instead:

    cat ./* > ../collection # CORRECT

    Prefixing relative globs with “./” always solves the “leading dash” problem, but it sure isn’t obvious. In fact, many shell books and guides completely omit this information, or don’t explain it until far later in the book (which many people never read). Even people who know this will occasionally forget to do it. After all, people tend to do things the “easy way” that seems to work, resulting in millions of programs that have subtle bugs (which sometimes lead to exploits).

    @DWheeler said:

    But filenames can include newlines!

    @DWheeler said:

    Oh, and don’t display filenames. Filenames could contain control characters that control the terminal (and X-windows), causing nasty side-effects on display. Displaying filenames can even cause a security vulnerability — and who expects printing a filename to be a vulnerability?!? In addition, you have no way of knowing for certain what the filename’s character encoding is, so if you got a filename from someone else who uses non-ASCII characters, you’re likely to end up with garbage mojibake.

    I'm keeping this link bookmarked for the next time someone tries to extol the virtues of the Linux CLI.



  •  I love the yellow box:

    In a well-designed system, simple things should be simple, and the “obvious easy” way to do simple common tasks should be the correct way. I call this goal “no sharp edges” — to use an analogy, if you’re designing a wrench, don’t put razor blades on the handles. Typical Unix/Linux filesystems fail this test — they do have sharp edges.

    I do love me some blooded hands when working with tools. Bring on the razors!



  • Well, it was inherited from the very beginning by almost every unix derivative. Changing it will break a lot of stuff and make many (old and growing) unix admins grunt a lot, so not good.


    The canonical solution is to receive the list of files as a \0 delimited list, either in another file , piped or via an already opened descriptor. I don't know how it works today but having a balloon tell me that I cannot use some characters wasn't certainly encouraging.



  • @spamcourt said:

    Well, it was inherited from the very beginning by almost every unix derivative. Changing it will break a lot of stuff and make many (old and growing) unix admins grunt a lot, so not good.


    The canonical solution is to receive the list of files as a \0 delimited list, either in another file , piped or via an already opened descriptor. I don't know how it works today but having a balloon tell me that I cannot use some characters wasn't certainly encouraging.

    He points out in the article that most of the problems can be fixed without violating the POSIX standard... Linux distributions just (by and large) haven't done it yet. (Although they are making steps in the right direction, mostly forced via. having to be compatible with OS X and Windows.)

    The author of that essay didn't get that 80% of the point of naming the directory in Windows "Documents and Settings" with the space was to smoke-out programs/scripts that were doing it wrong. I dunno why they've set it back to "Users", but to be fair "Users" is a better name.


  • ♿ (Parody)

    @blakeyrat said:

    @spamcourt said:
    Well, it was inherited from the very beginning by almost every unix derivative. Changing it will break a lot of stuff and make many (old and growing) unix admins grunt a lot, so not good.


    The canonical solution is to receive the list of files as a \0 delimited list, either in another file , piped or via an already opened descriptor. I don't know how it works today but having a balloon tell me that I cannot use some characters wasn't certainly encouraging.

    He points out in the article that most of the problems can be fixed without violating the POSIX standard... Linux distributions just (by and large) haven't done it yet. (Although they are making steps in the right direction, mostly forced via. having to be compatible with OS X and Windows.)

    Yes, and I don't think that spamcourt disagrees with you on that point (at least, his comment doesn't). Note that he didn't have anything to say about the standard, just backwards compatibility with existing implementations. Did backward compatibility stop being a reason to continue painful things? :-P

    @blakeyrat said:

    The author of that essay didn't get that 80% of the point of naming the directory in Windows "Documents and Settings" with the space was to smoke-out programs/scripts that were doing it wrong. I dunno why they've set it back to "Users", but to be fair "Users" is a better name.

    Is that really a Good Thing, though? He does point out that spaces in file names are problematic, and I would guess have caused more problems than all the rest of it put together (which are genuine WTFs in their own right, to be sure). Most of the issues seem likely to occur as the result of something malicious (the encoding issues excepted), while spaces in file names happen all the time in normal operation, and one careless mistake is enough to fail.



  • @boomzilla said:

    Yes, and I don't think that spamcourt disagrees with you on that point (at least, his comment doesn't).

    Well I assume the old pissy Unix admins would stop grunting if you pointed out there's nothing in the standard requiring that newline be stored in a filename. I didn't think about backwards-compatibility; I guess I just read that differently.

    @boomzilla said:

    Is that really a Good Thing, though?

    Which one? Naming the folder "Documents and Settings" or naming it to "Users"? If the latter, "Program Files" and "Program Files (x86)" are both still around to force programs to handle spaces correctly, so I think it's a non-issue. Besides, if a program stayed broken throughout the entire Windows 2000 and Windows XP eras, it's never going to be fixed at this point.

    Sadly a lot of software still gets this wrong. If you install the Amazon AWS admin tools (written in Java, natch-- you can tell because they don't fucking work) into "Program Files", they fail hard. This also isn't documented anywhere. God forbid I put program files in a folder named "Program Files"!

    @boomzilla said:

    and I would guess have caused more problems than all the rest of it put together (which are genuine WTFs in their own right, to be sure).

    As the essay mentions, in Linix/Unix/POSIX, all of the problems caused by spaces are due to space being listed as a delimiter (the IFS variable). If you remove space from the list of delimiters, and at the same time disallow tabs and newlines from file names, 99% of those problems vanish... which is exactly what he proposes they do.


  • Garbage Person

    @blakeyrat said:

    The author of that essay didn't get that 80% of the point of naming the directory in Windows "Documents and Settings" with the space was to smoke-out programs/scripts that were doing it wrong. I dunno why they've set it back to "Users", but to be fair "Users" is a better name.
    Because it pissed off every single person who ever had to type a file path by hand.


  • ♿ (Parody)

    @blakeyrat said:

    @boomzilla said:
    Is that really a Good Thing, though?

    Which one? Naming the folder "Documents and Settings" or naming it to "Users"? If the latter, "Program Files" and "Program Files (x86)" are both still around to force programs to handle spaces correctly, so I think it's a non-issue. Besides, if a program stayed broken throughout the entire Windows 2000 and Windows XP eras, it's never going to be fixed at this point.

    Sadly a lot of software still gets this wrong. If you install the Amazon AWS admin tools (written in Java, natch-- you can tell because they don't fucking work) into "Program Files", they fail hard. This also isn't documented anywhere. God forbid I put program files in a folder named "Program Files"!

    I think Users is a huge improvement aside from getting rid of the spaces. And yep, lot of programs still manage to get "Program Files" wrong (not just java...in fact, probably not even mainly java). It still happens a lot. Maybe not with "big" commercial programs (though I'm sure a lot of WTFy enterprisey apps still do), but even little one off tools. My point is just that it's easy to get it wrong, and by general usability concepts, is shouldn't be so easy to get wrong. I'm certainly not defending the people who do get it wrong, although I'd be surprised if there's anyone who never ever made a similar mistake at least once.

    At least in any Linux distro I've ever seen, there are no system directories with spaces. Seriously, did we really need "Program Files"? Wouldn't "Programs" have been just as descriptive, but saved zillions of mistakes?

    @blakeyrat said:

    @boomzilla said:
    and I would guess have caused more problems than all the rest of it put together (which are genuine WTFs in their own right, to be sure).

    As the essay mentions, in Linix/Unix/POSIX, all of the problems caused by spaces are due to space being listed as a delimiter (the IFS variable). If you remove space from the list of delimiters, and at the same time disallow tabs and newlines from file names, 99% of those problems vanish... which is exactly what he proposes they do.

    Yeah, the non-breaking space thing was an interesting idea, too. Still, outside of scripts, people still mess up the spaces thing all the time, on pretty much every OS / filesystem that allows them. And he was mainly focusing on shell scripting type stuff. IME, outside of the standard Windows directories, it's user documents that are most likely to contain spaces, but directories with spaces that cause the most problems (admittedly, I'm not a huge shell scripter).

    The "special" files / directories (briefly mentioned in the essay) on Windows have always seemed....weird to me.



  • @boomzilla said:

    My point is just that it's easy to get it wrong, and by general usability concepts, is shouldn't be so easy to get wrong.

    Well the solution to that is to stop typing in paths manually. Ever.

    @boomzilla said:

    I think Users is a huge improvement aside from getting rid of the spaces. And yep, lot of programs still manage to get "Program Files" wrong (not just java...in fact, probably not even mainly java). It still happens a lot. Maybe not with "big" commercial programs (though I'm sure a lot of WTFy enterprisey apps still do), but even little one off tools. My point is just that it's easy to get it wrong, and by general usability concepts, is shouldn't be so easy to get wrong. I'm certainly not defending the people who do get it wrong, although I'd be surprised if there's anyone who never ever made a similar mistake at least once.

    At least in any Linux distro I've ever seen, there are no system directories with spaces. Seriously, did we really need "Program Files"? Wouldn't "Programs" have been just as descriptive, but saved zillions of mistakes?

    Here's the issue: if Microsoft puts in a bunch of documentation that says "hey, paths can have spaces in them, you have to be able to handle that", but they don't actually put in any paths with spaces in them, developers won't heed the warning. To get developers to pay attention, you have to actually break their incorrect programs in one way or another. Previously, they've done that by using spaces in important folders, or by putting "Not Responding" in their window titles, or by obstructing ActiveX applets with dozens of security warning dialogs. More recently, they've done that by adding UAC warnings to programs that incorrectly use file permissions, or attempt to write into system folders.

    It's the only way to get programmers to stop writing shit software. Psychology is important.

    @boomzilla said:

    The "special" files / directories (briefly mentioned in the essay) on Windows have always seemed....weird to me.

    That's just legacy crap from DOS. They really should tear it out, but I guarantee there's a business somewhere that does millions of dollars of transactions through some program that opens "COM1".


  • ♿ (Parody)

    @blakeyrat said:

    @boomzilla said:
    My point is just that it's easy to get it wrong, and by general usability concepts, is shouldn't be so easy to get wrong.

    Well the solution to that is to stop typing in paths manually. Ever.

    No, it isn't. I mean, that's one way the problem manifests, but won't solve it. Not by a long shot.

    @blakeyrat said:

    osoft puts in a bunch of documentation that says "hey, paths can have spaces in them, you have to be able to handle that", but they don't actually put in any paths with spaces in them, developers won't heed the warning. To get developers to pay attention, you have to actually break their incorrect programs in one way or another. Previously, they've done that by using spaces in important folders, or by putting "Not Responding" in their window titles, or by obstructing ActiveX applets with dozens of security warning dialogs. More recently, they've done that by adding UAC warnings to programs that incorrectly use file permissions, or attempt to write into system folders.

    Sorry, but that's still bullshit from a usability standpoint. It's nice and all from a "tough love" perspective, but it's still problematic in the real world. If those directories didn't have spaces, then 99% of the problems caused by not using spaces would probably never have happened.

    What's the most common failure? It's gotta be when you start a process, either via an automated script or by using something like ShellExecute (or system()) and forgetting to quote your arguments. This would still bite you when you tried to pass a document with spaces in a similar command line, but most use of documents would be used via OpenFile or similar, where you don't have to worry about spaces.

    Your other examples are in a totally different category. They can be justified in and of themselves.

    @blakeyrat said:
    It's the only way to get programmers to stop writing shit software. Psychology is important.

    And yet it still doesn't stop it.



  • @blakeyrat said:

    Sadly a lot of software still gets this wrong. If you install the Amazon AWS admin tools (written in Java, natch-- you can tell because they don't fucking work) into "Program Files", they fail hard. This also isn't documented anywhere. God forbid I put program files in a folder named "Program Files"!

     See if you can install them in Progra~1 instead.

     



  • @boomzilla said:

    If those directories didn't have spaces, then 99% of the problems caused by not using spaces would probably never have happened.

    I would argue that they still would have happened, they just would have happened after the software was already shipped and it might be too late to fix it. By using "Program Files" on all computers, including those used by the developers of the program, at least the program will fail quickly and be fixed quickly, long before it's shipped to customers.

    @boomzilla said:

    What's the most common failure? It's gotta be when you start a process, either via an automated script or by using something like ShellExecute (or system()) and forgetting to quote your arguments.

    You wouldn't "forget" to quote your arguments if you never, ever manually typed in a path ever. So I stand by my solution.

    @boomzilla said:

    @blakeyrat said:
    It's the only way to get programmers to stop writing shit software. Psychology is important.

    And yet it still doesn't stop it.

    Yes, well, it's not like we can run a proper A/B test on this. There's no really good way to tell if those tactics work, or if the software would have improved on its own without them.



  • @blakeyrat said:

    @DWheeler said:
    But filenames can include newlines!

    I ones had filenames containing slashes. Dunno if it was a bug of something, but I really had that! They were encoded as %2F in the filename, but displayed as slashes. Couldn't reproduce it right now though. Probably the bug was fixed.


  • ♿ (Parody)

    @blakeyrat said:

    @boomzilla said:
    If those directories didn't have spaces, then 99% of the problems caused by not using spaces would probably never have happened.

    I would argue that they still would have happened, they just would have happened after the software was already shipped and it might be too late to fix it. By using "Program Files" on all computers, including those used by the developers of the program, at least the program will fail quickly and be fixed quickly, long before it's shipped to customers.

    Yeah, it probably did save a few like that. Except that, like I said, the majority of the problem cases were manufactured by having "Program Files" or "Documents and Settings" in the first place! And still plenty of them did and still do ship without doing this right. Remember your startup "C:\Program" WTF?

    @blakeyrat said:

    @boomzilla said:
    What's the most common failure? It's gotta be when you start a process, either via an automated script or by using something like ShellExecute (or system()) and forgetting to quote your arguments.

    You wouldn't "forget" to quote your arguments if you never, ever manually typed in a path ever. So I stand by my solution.

    WTF are you talking about? Who says that you're typing a path in manually? You've got a string, somehow, maybe it came from a dialog, or from looking through a directory, or from the registry. Who cares? Now you try to use it, and guess what, no one typed it in manually, but it still fails when you use it in a way that needs to be quoted, but wasn't.

    @blakeyrat said:

    @boomzilla said:
    @blakeyrat said:
    It's the only way to get programmers to stop writing shit software. Psychology is important.

    And yet it still doesn't stop it.

    Yes, well, it's not like we can run a proper A/B test on this. There's no really good way to tell if those tactics work, or if the software would have improved on its own without them.

    I agree. BTW, do you have any sort of reference that this was the actual motivation for putting spaces in those directories? Still, when the majority of space mishandling is done when using system directories (yes, it's an assertion based only on my experience), you should admit that maybe you made the problem worse.



  • @boomzilla said:

    WTF are you talking about? Who says that you're typing a path in manually? You've got a string, somehow, maybe it came from a dialog, or from looking through a directory, or from the registry. Who cares? Now you try to use it, and guess what, no one typed it in manually, but it still fails when you use it in a way that needs to be quoted, but wasn't.

    Hm. I guess? I can't think of a case like that off the top of my head, though.

    @boomzilla said:

    Still, when the majority of space mishandling is done when using system directories (yes, it's an assertion based only on my experience), you should admit that maybe you made the problem worse.

    Why should I admit it? I had nothing to do with it.

    But either way, since you can't prove your assertion that it made things worse, and I can't prove my assertion that it made things better, it seems like a stalemate to me.



  • Fun fact: older windows versions actually have localized folder names for things like program files. Swedish just used "program". Guess what happened when people forgot to escape their spaces when incorrect using the non localized name?

    Sad fact: localization in new systems messes with the display names of folders instead. The shell says that it's called "My documents", but the filesystem says that it is called "Documents".


  • ♿ (Parody)

    @henke37 said:

    Sad fact: localization in new systems messes with the display names of folders instead. The shell says that it's called "My documents", but the filesystem says that it is called "Documents".

    There are all sorts of weird things that go on now inside of the Users folder. Was trying to migrate my wife from Vista to a new 7 machine, and needed to bring over her Outlook junk. There appears to be no way to navigate there using Explorer. The only way was to Win+R, then paste in some folder name, which would open explorer in the proper place. But trying to navigate there manually through explorer's tree got me (as admin) a "Cannot access" one of the dirs on the way there.

    Also, it's a Home Premium install, but there's all sorts of "Roaming" junk in the users' folders. WTF? I thought the point of all the ridiculous levels was to get rid of stuff you didn't need. So, you end up with Roaming profile dirs, but cannot RDP into it. Something that's actually useful for a home user.



  • @boomzilla said:

    But trying to navigate there manually through explorer's tree got me (as admin) a "Cannot access" one of the dirs on the way there.

    Explorer has issues with reparse points. If you find out the real name of the folder, you can navigate there normally.

    @boomzilla said:

    Also, it's a Home Premium install, but there's all sorts of "Roaming" junk in the users' folders. WTF? I thought the point of all the ridiculous levels was to get rid of stuff you didn't need. So, you end up with Roaming profile dirs, but cannot RDP into it. Something that's actually useful for a home user.

    Roaming is where you put stuff that should be roamed, whether or not roaming profiles is turned on or not.


  • Garbage Person

    @blakeyrat said:

    Explorer has issues with reparse points. If you find out the real name of the folder, you can navigate there normally.
    TRWTF is that these issues made production.



  • @Weng said:

    @blakeyrat said:

    Explorer has issues with reparse points. If you find out the real name of the folder, you can navigate there normally.
    TRWTF is that these issues made production.

    Yeah. It was never really an issue before the OS actually started using reparse points... now it leads to all kinds of crazy stuff. Like the people who think WinSxS actually takes up 12 GB.



  • Interesting article.  Looks to me basically like the Unix/C "worse is better" philosophy in action: It's more important to make a system that's easy to implement than one that actually works well, because at least then you have a finished product, and any deficiencies in your design can simply be passed downstream for someone else to deal with when they try to use it.


  • ♿ (Parody)

    @blakeyrat said:

    Roaming is where you put stuff that should be roamed, whether or not roaming profiles is turned on or not.

    I guess the question (TRWTF?) is why it's even there at all, since presumably there's no way to even turn it on in Home Premium.



  • @boomzilla said:

    @blakeyrat said:
    Roaming is where you put stuff that should be roamed, whether or not roaming profiles is turned on or not.

    I guess the question (TRWTF?) is why it's even there at all, since presumably there's no way to even turn it on in Home Premium.

    What do you expect to happen? Windows says "put files that need to roam in this folder unless you're running on Home Premium, in which case put them in this totally different folder because we want to make your life hard."


  • ♿ (Parody)

    @blakeyrat said:

    What do you expect to happen? Windows says "put files that need to roam in this folder unless you're running on Home Premium, in which case put them in this totally different folder because we want to make your life hard."

    I have no idea, really (which was obvious). They neuter stuff, supposedly so that users don't get confused by features they don't need (at least, that's the most common explanation that I've heard). I'm assuming that home users don't get actual roaming profiles. I have no clue about the infrastructure required to actually support whatever roams. I just know that encountering different Local and Roaming and whatever junk is in there--especially when it seems to be in both--is really confusing to a user who doesn't understand the ins and outs of Windows profile management (which I assume is most, even for enterprise editions).



  • @boomzilla said:

    I just know that encountering different Local and Roaming and whatever junk is in there--especially when it seems to be in both--is really confusing to a user who doesn't understand the ins and outs of Windows profile management (which I assume is most, even for enterprise editions).

    Why is that user even looking in there? This is the part that is not adding up for me.



  • The reason a user would look into the roaming folder is because some stupid software is installed,with no user input, there and to do anything useful with it the user needs to get into the install directory.


  • Garbage Person

    @delta534 said:

    The reason a user would look into the roaming folder is because some stupid software is installed,with no user input, there and to do anything useful with it the user needs to get into the install directory.
    *coughGOOGLEPRODUCTScough*

     

    This is, however, Google's fucking failure, rather than anything in any way related to Microsoft.


  • ♿ (Parody)

    @blakeyrat said:

    @boomzilla said:
    I just know that encountering different Local and Roaming and whatever junk is in there--especially when it seems to be in both--is really confusing to a user who doesn't understand the ins and outs of Windows profile management (which I assume is most, even for enterprise editions).

    Why is that user even looking in there? This is the part that is not adding up for me.

    As I mentioned in my previous post, I was trying to migrate my wife from one machine to another. That means moving profile information for various apps (browsers, Outlook, etc). Some browsers, like Opera, at least, will tell you exactly where they store stuff. It took a fair amount of googling to figure out how to find the files for Outlook.



  • So BASH is TRWTF. I agree 100% and you really really really shouldn't use it unless you KNOW your filenames won't contain anything more exotic than spaces. Fortunately, we have alternatives like perl, python, ruby, php and others I forget - all languages that do not rely on function output automatically being interpreted as program source code. For every posix shell tool there's an analogous function in your favourite non-crap programming language to do the job without caveats.


    The answer to every "how do I ... in bash correctly", is simply "DO NOT FUCKING USE FUCKING BASH!"



  • @blakeyrat said:

    The author of that essay didn't get that 80% of the point of naming the directory in Windows "Documents and Settings" with the space was to smoke-out programs/scripts that were doing it wrong.
    That was Program Files actually. Documents and Settings came much later (it wasn't there in Win95 and NT4, both of which had the user profiles in Windows directory).@blakeyrat said:
    I dunno why they've set it back to "Users", but to be fair "Users" is a better name.
    AFAIK, one of the reasons was that programs were hitting the MAX_PATH limit.

    @blakeyrat said:

    Explorer has issues with reparse points.
    It doesn't - it's just that the compatibility reparse points in Vista and 7 are specifically crafted so you can't navigate to them (but you can navigate to a directory in them). This is to prevent old backup software from backing up certain things twice (and in some cases from going into loops), while keeping most of the old programs that accessed hardcoded paths working. Instead of going to C:\Users\username\Application Data, you should go to C:\Users\username\AppData\Roaming (Local Settings\Application Data is AppData\Local).

    BTW, Outlook 2010 stores it's PST files in (My) Documents by default. Which is really fun when you've got them redirected to a network share.


  • ♿ (Parody)

    @ender said:

    BTW, Outlook 2010 stores it's PST files in (My) Documents by default. Which is really fun when you've got them redirected to a network share.

    No, it doesn't:

    @Move Outlook information to another computer that has Outlook 2010 installed said:

    Step 1: Copy the files from the old computer

    On your old computer, exit Outlook. Open the folder where your .pst file is saved. These steps are for the default location. If you created or moved your data files to another folder, open that folder.

    Because the default folder is hidden folder, the easiest way to open the folder is to use the command %USERPROFILE%\Local Settings\Application Data\Microsoft\Outlook on the Start menu.

    Windows 7 Click Start. Next to the Shut down button, in the Search programs and files box, type %USERPROFILE%\Local Settings\Application Data\Microsoft\Outlook and then press Enter.



  • @Mo6eB said:

    So BASH is TRWTF. I agree 100% and you really really really shouldn't use it unless you KNOW your filenames won't contain anything more exotic than spaces. Fortunately, we have alternatives like perl, python, ruby, php and others I forget - all languages that do not rely on function output automatically being interpreted as program source code. For every posix shell tool there's an analogous function in your favourite non-crap programming language to do the job without caveats.


    The answer to every "how do I ... in bash correctly", is simply "DO NOT FUCKING USE FUCKING BASH!"

    The article mentions that the guys who made git re-wrote several shell operations in C, so that they could correctly and easily handle weird file names. He also mentions that if a group of your best developers need to be rewriting your shell operations in a different language, your shell is shit.

    I mean what you're saying is all fine and good, but we still frequently see people on this forum and others going, "everybody should be using Linux because the CLI is so awesome you can do everything in the CLI the CLI bought me a lotto ticket once and it won $50 etc etc". Which is why I'm saving this bookmark, because when experts on the OS say it's shit, that means all those people are full of shit. And I like pointing out when people are full of shit.

    @ender said:

    @blakeyrat said:
    Explorer has issues with reparse points.
    It doesn't - it's just that the compatibility reparse points in Vista and 7 are specifically crafted so you can't navigate to them (but you can navigate to a directory in them). This is to prevent old backup software from backing up certain things twice (and in some cases from going into loops), while keeping most of the old programs that accessed hardcoded paths working. Instead of going to C:\Users\username\Application Data, you should go to C:\Users\username\AppData\Roaming (Local Settings\Application Data is AppData\Local).

    While your explanation about why the permissions on the reparse points are set the way they are is true, you're dead wrong saying that Explorer can handle them correctly. (Those two things aren't mutually-exclusive, you know.) The WinSXS directory is a perfect example.



  • @blakeyrat said:

    The article mentions that the guys who made git re-wrote several shell operations in C, so that they could correctly and easily handle weird file names. He also mentions that if a group of your best developers need to be rewriting your shell operations in a different language, your shell is shit.
     

    Hmm?  What language is the shell written in, then?

     



  • @Mason Wheeler said:

    @blakeyrat said:

    The article mentions that the guys who made git re-wrote several shell operations in C, so that they could correctly and easily handle weird file names. He also mentions that if a group of your best developers need to be rewriting your shell operations in a different language, your shell is shit.
     

    Hmm?  What language is the shell written in, then?

     

    Are you just getting out of the advanced course in point-missing? Point-missing 301? And its lab course: Point-missing 321?

    The article gives examples of how it's virtually impossible to write a few extremely simple shell operations using the built-in commands. One of those is traversing every file in a folder. The article demonstrates about 20 different ways of doing this, then explains why every single one either 1) isn't portable (relies on stuff not in the POSIX standard), 2) doesn't work right for all possible filenames.

    Based on that article, is it any surprise they would write a C program to traverse a directory? The alternatives is having known heisenbugs in their code ("ok this works assuming A, and assuming B, and assuming C, otherwise it fails hard"), or makes their product non-portable. The C alternative, on the other hand, can handle filenames with control characters, newlines, tabs, etc. and remains portable because it can be written without going outside what POSIX allows.

    (I guess "known heisenbugs" is a oxymoron, but you get the point.)


  • ♿ (Parody)

    @blakeyrat said:

    I mean what you're saying is all fine and good, but we still frequently see people on this forum and others going, "everybody should be using Linux because the CLI is so awesome you can do everything in the CLI the CLI bought me a lotto ticket once and it won $50 etc etc". Which is why I'm saving this bookmark, because when experts on the OS say it's shit, that means all those people are full of shit. And I like pointing out when people are full of shit.

    The problem with this line of argument is that it's effectively equivalent to saying that you should never use windows because of malware infections. Why? As mentioned in the article, the biggest offenders of this sort of thing are malicious intent.

    I think that Mason was really on to something with the "worse is better" philosophy. Obviously, there are other legitimate points of view. But maybe you're right. For the reasons in this article, nothing productive ever happens via CLI on Linux. That's gotta be it.

    Also, I'd like to see some of these people saying "everybody should be using Linux because the CLI is so awesome."



  • @boomzilla said:

    Also, I'd like to see some of these people saying "everybody should be using Linux because the CLI is so awesome."

    There was one on this very forum not 3 months ago. I should see if I can dig up the thread.


  • ♿ (Parody)

    @blakeyrat said:

    @boomzilla said:
    Also, I'd like to see some of these people saying "everybody should be using Linux because the CLI is so awesome."

    There was one on this very forum not 3 months ago. I should see if I can dig up the thread.

    You should. I remember discussing CLIs in that time frame. But I don't remember statements like yours. I do remember people saying that often they are very productive and often prefer CLIs to GUIs. I guess they'll have to revise their experience after reading this thread, huh?



  • Yes. Yes they will.



  • @boomzilla said:

    But I don't remember statements like yours. I do remember people saying that often they are very productive and often prefer CLIs to GUIs. I guess they'll have to revise their experience after reading this thread, huh?
    About a month ago i came across an article about CLI vs GUI.  I can't find it now, but the interesting part was that people who use a CLI frequently swear that they are able to work faster than using a GUI,  but timing tests consistantly show that people can perform tasks faster using a GUI.  It's not that a CLI is bad or unneccessary, but the idea of being "more productive when using a CLI" is more perception than reality.



  • @blakeyrat said:

    I just came across this awesome essay explaining just how difficult dealing with POSIX filenames is. I'm not even all the way through, and the WTF quantity is very high.
     

    Filenames can contain control characters, including newlines.

     Holy shit.  Seriously.  What kind of retard came up with that idea?



  • @boomzilla said:

    NoYes, it doesn't
    FTFY.@Move Outlook information to another computer that has Outlook 2010 installed said:
    Note     Beginning with Outlook 2010, all new Outlook Data Files (.pst) are created in your Documents folder in a new folder named Outlook Files. If you upgraded your old computer to Outlook 2010, you could have .pst files in both locations. Make sure that you check both locations if you were using Outlook 2010 on the old computer.



  • @blakeyrat said:

    The author of that essay didn't get that 80% of the point of naming the directory in Windows "Documents and Settings" with the space was to smoke-out programs/scripts that were doing it wrong.

    []citation needed]

     



  • @El_Heffe said:

    @blakeyrat said:
    I just came across this awesome essay explaining just how difficult dealing with POSIX filenames is. I'm not even all the way through, and the WTF quantity is very high.
     

    Filenames can contain control characters, including newlines.

    Holy shit.  Seriously.  What kind of retard came up with that idea?

    See my previous post re: "worse is better" philosophy.  It's not "an idea that someone came up with;" it's simply that it's easier to not write code implementing a restriction than is is to write that code.  And then when things go wrong, oh well, it was the downstream user's fault, not the person who designed the system.  The downstream user should have known better.

    This is one of the core philosophical principles of C and Unix, and in a benevolent world, it would be more or less viable.  But the idea doesn't account for the existence of malicious downstream users.  We've known it was a failed idea since 1989, but people are still following it.

     



  • "garbage mojibake" is redundant. Shitty essay.



  • @Mason Wheeler said:

    This is one of the core philosophical principles of C and Unix, and in a benevolent world, it would be more or less viable.  But the idea doesn't account for the existence of malicious downstream users.  We've known it was a failed idea since 1989, but people are still following it.

     

    Doesn't account for the existence of malicious user? B-b-but, I thought Unix was InherentlySecure. I heard it was DesignedForSecurityFromTheGroundUp.


  • BINNED

    @El_Heffe said:

    About a month ago i came across an article about CLI vs GUI.  I can't find it now, but the interesting part was that people who use a CLI frequently swear that they are able to work faster than using a GUI,  but timing tests consistantly show that people can perform tasks faster using a GUI.  It's not that a CLI is bad or unneccessary, but the idea of being "more productive when using a CLI" is more perception than reality.
    Are the subjects of the timing tests the same people who swear the CLI is faster, or are they the general population?


  • BINNED

    @boomzilla said:

    Also, I'd like to see some of these people saying "everybody should be using Linux because the CLI is so awesome."

    Actually, I'd like to see less people saying that. The CLI is definitely not for everyone.


  • @Mo6eB said:

    So BASH is TRWTF. I agree 100% and you really really really shouldn't use it unless you KNOW your filenames won't contain anything more exotic than spaces. Fortunately, we have alternatives like perl, python, ruby, php and others I forget - all languages that do not rely on function output automatically being interpreted as program source code. For every posix shell tool there's an analogous function in your favourite non-crap programming language to do the job without caveats.

    The answer to every "how do I ... in bash correctly", is simply "DO NOT FUCKING USE FUCKING BASH!"

     

    It's not as nice as a proper programming language, but you can at least rely on it being installed on any Linux system (sorry, I don't know or care what POSIX requires). If your installer relies on perl and File::Find to work, but your program does not, it may be time for you to learn how to use sh instead.

    While I don't know what the article author was trying to do when he encountered the problems described (alas, blakeyrat isn't as good at quoting as bash is), I do know that "ls . | processing_command" is generally worse than "find . -maxdepth 1 -mindepth 1 -execdir processing_command \;", even if it's quicker to type in a script.

    It is unfortunate that the command to do exactly what you want isn't always the first one you think of which sort-of works, but that's just the way it goes.



  • @Mo6eB said:

    So BASH is TRWTF. I agree 100% and you really really really shouldn't use it unless you KNOW your filenames won't contain anything more exotic than spaces. Fortunately, we have alternatives like perl, python, ruby, php and others I forget - all languages that do not rely on function output automatically being interpreted as program source code. For every posix shell tool there's an analogous function in your favourite non-crap programming language to do the job without caveats.

    The answer to every "how do I ... in bash correctly", is simply "DO NOT FUCKING USE FUCKING BASH!"

     

    It's not as nice as a proper programming language, but you can at least rely on it being installed on any Linux system (sorry, I don't know or care what POSIX requires). If your installer relies on perl and File::Find to work, but your program does not, it may be time for you to learn how to use sh instead.

    While I don't know what the article author was trying to do when he encountered the problems described (alas, blakeyrat isn't as good at quoting as bash is), I do know that "ls . | processing_command" is generally worse than "find . -maxdepth 1 -mindepth 1 -execdir processing_command \;", even if it's quicker to type.

    It is unfortunate that the command to do exactly what you want isn't always the first one you think of, but that's just the way it goes.



  • @El_Heffe said:

    @boomzilla said:
    But I don't remember statements like yours. I do remember people saying that often they are very productive and often prefer CLIs to GUIs. I guess they'll have to revise their experience after reading this thread, huh?
    About a month ago i came across an article about CLI vs GUI.  I can't find it now, but the interesting part was that people who use a CLI frequently swear that they are able to work faster than using a GUI,  but timing tests consistantly show that people can perform tasks faster using a GUI.  It's not that a CLI is bad or unneccessary, but the idea of being "more productive when using a CLI" is more perception than reality.

    It really really depends on the task. Without any further information or the article you can't really support the claim. typing commands in general is probably slower/about as fast as using a gui for a lot of tasks, but commands have the potential to do a lot of work in a single command and are really flexible while GUI's are very stiff and always do a moderate amount of work per action.

    A real-life example that I had to do a few weeks ago: a overview of all mailing lists on a server. This costs me about 30 seconds on the command line (including thinking what I wanted to do):

    ls /some/directory/with/mailinglists | mail -s "Mailing list report" someAdress@example.org
    

    Using GUI's this wouldn't have been hard either, but would still take significantly longer: opening an file explorer, copying the list of files, opening your mail app, starting a new mail, pasting it in your mail , sending the mail. You're not telling me that that's faster.

    My point being that CLI's can really shine in specific situations even if they're about as fast or slower in other situations.


Log in to reply