🐧 Lunix



  • The most educational part is the addendum:

    Update: added -print0 to find and -0 to xargs to properly handle spaces in file names.
    

    Firstly, this is a really dangerous class of bug. Unsafe handling of spaces in filenames is the kind of shell scripting mistake that will eventually end up deleting half the files on your computer when you just wanted to prune a directory.



  • Meh. Just another n00b completely misunderstanding "the" (or "a") UNIX philosophy.

    Having one program per task that does its task well does not mean that programming magically becomes easy. Other important benefits include modularity of code (in days when everything was C or assembly and not object oriented), and the ability to understand each piece very well. (Think from a sys-admin perspective who can understand that grep, sort, and uniq all do exactly what they are told and nothing more.)

    And in the case of shell scripts, you don't need -print0 or -0 to handle spaces in file names. That is more for unicode characters and other odd cases. In Bash setting IFS=$'\n' works just fine to split the output of a program by newlines only and not spaces.

    Like any language, you can mess yourself up in shell scripts if you don't think and understand the limitations of piping one program to another. You need split file names by newlines or null characters, quote file names that are passed as arguments to programs that you call, etc. No different than knowing that you need to use parametrized statements in SQL.



  • @blakeyrat said:

    Unsafe handling of spaces in filenames is the kind of shell scripting mistake that will eventually end up deleting half the files on your computer when you just wanted to prune a directory.

    In order for that to be true in this case, you would need to have a jpeg with a filename that starts with a space. GUI file managers (for any OS) won't let you do that.



  • Or you could use PowerShell, which has actual objects and if you pipe a file descriptor around it remains a file descriptor and doesn't have to be serialized and deserialized into text 47,236 times.

    But then again, PowerShell was actually designed by smart people, and not just glommed-together by idiots.



  • Can it list the files in a directory recursively without touching paths?

    Also, what the hell do you think < and > and >> and | and <(foo) and >(foo) do if not "work with file descriptors"?



  • ... I have no idea what "touching paths" means, so the answer is either "no", or "of course".



  • Basically, I want to do something on all the files in a deep directory tree and I don't want to encounter the MAX_PATH bug that exists on Windows for backwards compatibility.



  • The point is, in the Linux example, when you pipe a list of files from one tool to another, the only way of doing that is by piping a list of paths-- which brings up the issues the article talks about, both tools have to agree about the same way of delimiting the paths, indicating the list is over, handling "special" characters like dashes or unicode or non-printables, etc. And if they don't agree you get nasty side-effects with potentially disastrous effects.

    If a PowerShell tool pipes a list of files to another PowerShell tool, it's a list of file objects. List is a standard type, and a FileInfo is a standard type, and every PowerShell tool knows exactly how they work so there's no issues-- no matter what funky characters the files may have in their names.

    The Linux way is, "everything's text". Which is completely, obviously, wrong. A list of files isn't text, it shouldn't be treated like text, or you end up with all kinds of shitty results. Combine that with the particular Linux disease of allowing virtually any character into a filename (including characters like carriage returns!) and you end up with potential disaster.



  • @ben_lubar said:

    Basically, I want to do something on all the files in a deep directory tree and I don't want to encounter the MAX_PATH bug that exists on Windows for backwards compatibility.

    Ok? And?

    I have absolutely no idea what you're talking about, and you still haven't defined what "touching paths" means. Does the FileInfo class include a path as one of its member variables? Almost certainly. Does it use it to find child directories? I honestly don't know, but I doubt it. Do you need to know the path to open a FileStream given a FileInfo? Not at all. Does the data simply being present count as "touching" it? What does "touching" mean?

    EDIT: is it affected by the MAX_PATH issue? Again, I don't know but I doubt it.



  • So if I don't have spaces in my filenames, it's all good, right?

    (I actually wouldn't mind PowerShell on Linux, PowerShell is actually pretty decent under the hood...)



  • @tar said:

    So if I don't have spaces in my filenames, it's all good, right?

    If you don't have file names with spaces, and also don't have any characters that either tool might consider a delimiter (including, most importantly, carriage return but also characters like comma or tab), and none of the file names start with a dash and none of them are a single asterisk... you might be good. Maybe. But I wouldn't count on it.

    A better solution would be the OS having a standard library which is use for every tool that passes a list of filenames to another tool. An even better solution would be to have that, but also prevent the user from saving files that are known to be incompatible with the above-mentioned library. Of course, an even more better solution would be to not treat a list of files as just a string of text in the first place. Shockingly, Windows does all of those. Linux does zero of them.



  • @blakeyrat said:

    Of course, an even more better solution would be to not treat a list of filenames as just a string of text in the first place

    @blakeyrat said:

    Shockingly, Windows does all of those

    Windows filenames aren't strings of text?



  • @ben_lubar said:

    Windows filenames aren't strings of text?

    If you keep all your data in a single text file on the desktop (SSDS00.TXT), you can avoid these issues. It's also very convenient for searching as well, especially if you have some videos embedded in there.



  • s/PowerShell/Python/ and everything is an object there.



  • Filenames are strings of text. Files are not. I typed the wrong thing.



  • I find it fascinating that linux has such a problem with whitespaces in paths. Some articles are even suggesting not using it because it might break shell scripts easily. The only problem: Whitespace is probably the most important "special" character you can have: Everyone and their dog uses it to name files. Probably all other characters don't need to appear in a filename, but a fucking space is clearly a must.



  • @ben_lubar said:

    Basically, I want to do something on all the files in a deep directory tree and I don't want to encounter the MAX_PATH bug that exists on Windows for backwards compatibility.

    This is probably the bug i hit most om Windows. God i hate it. Mostly when trying to store code/webpages in a sane place. Now all code just lives in single or dual letter folders in the root. ('Vs', 'web','a','aa' etc.)



  • They certainly aren't NUL-delimited.

    UNIX filenames have two restrictions:

    • Path components (directory names and file names) cannot contain the / character (0x2F)
    • The path cannot contain the NUL character (0x00)

    The rules are pretty simple to understand...


  • Discourse touched me in a no-no place

    @riking said:

    UNIX filenames have two restrictions:

    A filesystem might impose additional restrictions, e.g., if it was being backed by something that was also intended to be used by Windows as well.


  • FoxDev

    @blakeyrat said:

    But then again, PowerShell was actually designed by smart people, and not just glommed-together by idiots.

    Powershell is also far newer, and designed as a ground-up replacement for CMD. Saying it's better is a bit like saying a Ferrari is faster than a horse: true, but also an unfair comparison.



  • This just just shows your total ignorance on what you're writing about.

    1. This not Linux (kernel, remember?)
    2. Bash? Maybe ksh handles piping differently
    3. find/xargs are just applications present in any *nix. And usually, when you find them together is a hack... Your doing it wrong™
    4. For 30 years, using spaces in filenames has been a bad idea in *nix systems. That you don't know already to workaround those is another reason to stay away.

    So, let's translate this stupid rant to your understanding. "Windows sucks because when I drag a SVG from the desktop to MSIE it opens the XML buuuaaahhh!!"

    not sure if this still happens, but it did when MSIE didn't support SVG



  • @ben_lubar said:

    a jpeg with a filename that starts with a space. GUI file managers (for any OS) won't let you do that.



  • Powershell is also far newer, and designed as a ground-up replacement for CMD. Saying it's better is a bit like saying a Ferrari is faster than a horse: true, but also an unfair comparison.

    And yet, nobody in the Linux world has any intention to develop a replacement for the shell. The shell is our Lord and saviour, protector of the Unix Way, and doubting It is not allowed. Try saying that it sucks in any Linux forum and see what happens.


  • FoxDev

    @anonymous234 said:

    Try saying that it sucks in any Linux forum and see what happens.

    If I want to commit seppuku, I'll do it the traditional way, TYVM ;)


  • kills Dumbledore

    @Eldelshell said:

    For 30 years, using spaces in filenames has been a bad idea in *nix systems.

    Is that meant to be an endorsement of *nix? Because spaces in filenames are useful and an OS that doesn't support them properly should think about supporting them properly


  • FoxDev

    It's not really a Linux issue though; it's more a filesystem/shell issue. But given how many shells and filesystems have been written since 1066, it's surprising no-one's thought "Hey, maybe we should, y'know, improve things, rather than make the same mistakes over and over again…"



  • That's probably due to the fact that shell-lover are as conservative as they come. Additionally, there are a lot of CLI programs, so changing the interface is realy hard.
    PowerShell is gaining traction, but it's slow going, not least because it is more complicated than understanding the basics of the CLI.


  • Winner of the 2016 Presidential Election

    @blakeyrat said:

    Combine that with the particular Linux disease of allowing virtually any character into a filename (including characters like carriage returns!) and you end up with potential disaster.

    That's not the :wtf:, the :wtf: is that there are people who still use shell scripts in 2015. Hell, there are even quite a few idiots who think systemd is bad because it abolishes those fragile, "easily debuggable" shell scripts which were controlling the system startup before.

    Also, NTFS isn't much better in this regard. It allows file names which no single tool on Windows will ever be able to do anything with. Once you've managed to create a file with a question mark, colon or quotation mark in its name, you won't even be able to rename it or delete it. Discoursistent!


  • BINNED

    I'd say this is slightly off: new shells are being written, they are just not being widely adopted. Just the other day I found about this: http://rubyshell.sourceforge.net.

    Does it suffer from same problems? Honestly, I don't know. But the point is someone made one. Have you heard about it? Probably not, because it's not installed by default on any distro.

    PowerShell has the advantage of being made by Microsoft, using Microsoft's languages, for Microsoft's operating systems. Of course people will stumble onto it and start using it. On Linux, even if I sat down and wrote my own shell that solves all these issues right now, how much adoption would it get? Maybe 5 people would find it and use it.

    The only way I see some progress being made if Debian, or RHEL, or some other big player made and / or adopted something like this and made it the default on their distro.

    As I was writing this @asdf just mentioned systemd which is actually a fine demonstration of this point - no one gave a fuck about it until Debian adopted it. Slashdot exploded, accusations are being flung, but you know what? I use systemd right now. Sure, 99% of my services use the compatibility layer for systemv, and I still use service x restart instead of systemctl restart x, but it's there. Same could be done for bash. There would be kicking and screaming, but it would be doable.

    The question is - does any of the big distros even want to try something like that?


  • Winner of the 2016 Presidential Election

    @Onyx said:

    Same should be done for bash. There would be kicking and screaming, but it would be doable.

    FTFY.

    BTW: Does anyone know what happened to kmscon?


  • FoxDev

    @Onyx said:

    PowerShell has the advantage of being made by Microsoft, using Microsoft's languages, for Microsoft's operating systems.

    And MS are pushing it hard as a replacement for CMD



  • @Onyx said:

    Sure, 99% of my services use the compatibility layer for systemv, and I still use service x restart instead of systemctl restart x, but it's there.

    I HATE the argument order switchover. What the fuck where they thinking? If I'm working with service x, I might want to do service x status, then service x stop then tinker with conf, then service x start. With systemctl, I constantly have to edit around their stupid argument order. Also type a lot more (blah.service instead of just blah).

    Therefore, I think sytemd authors should be burned and everything reverted back to good old 17th century sysvinit technology.

    Also, blakeyrat mistyped the title. Typical brainless Gates fanboy. World is full of idiots these days.


  • BINNED

    @cartman82 said:

    (blah.service instead of just blah)

    OBJECTION! I never wrote blah.service (unless I wanted to edit the actual file) and it worked. Thy system be poorly configured, sir!

    @cartman82 said:

    Therefore, I think sytemd authors should be burned and everything reverted back to good old 17th century sysvinit technology.

    I'll go with you on the first part (who doesn't like a good burning?) and agree to debate you about the latter (with possible burning afterwards).

    @cartman82 said:

    Also, blakeyrat mistyped the title. Typical brainless Gates fanboy. World is full of idiots these days.

    He just thought about the wrong platform. It happens.



  • Well, it's also a Unix with a CLI ...


  • Java Dev

    The problem with 'processes communicating filenames with each other' isn't a shell problem. Some of the problems arising around it are shell problems (like the way various shell expansions work). But some of them are in the individual tools. Like how there's no simple way to determine if a string of text is a filename or a program argument. And the fact pipes work in plain text without any indication of content-type.



  • @blakeyrat said:

    EDIT: is it affected by the MAX_PATH issue? Again, I don't know but I doubt it.
    It is, since .NET runs on top of the Win32 subsystem, which for compatibility constraints can't ever increase MAX_PATH. If they made a .NET NT, though, then it wouldn't be.@swayde said:
    This is probably the bug i hit most om Windows. God i hate it. Mostly when trying to store code/webpages in a sane place. Now all code just lives in single or dual letter folders in the root.
    The hell? What on earth could you possibly be doing that you hit it with any frequency at all? And your "solution" is dumb and you should feel bad.@asdf said:
    Also, NTFS isn't much better in this regard. It allows file names which no single tool on Windows will ever be able to do anything with. Once you've managed to create a file with a question mark, colon or quotation mark in its name, you won't even be able to rename it or delete it. Discoursistent!
    Again, that's the dumb idiot retarded Win32 personality getting in your way. If you use an app that's native NT or uses POSIX (and no, not mSys or Cygwin since those foolishly use dumb idiot retarded Win32), they're immune from that problem. Also, how are you able to create those files from Windows (without those tools) in the first place?


  • Winner of the 2016 Presidential Election

    @TwelveBaud said:

    Also, how are you able to create those files from Windows (without those tools) in the first place?

    Honestly: I have no idea. But I encountered those files on my external, NTFS-formatted HDD yesterday and had to copy them to my Laptop via a Ubuntu VM. I don't really care whose fault this is, but Windows not being able to read files from a file system made specifically for Windows - while having no problem displaying those files in the Explorer - is a WTF. Especially since the copy operation failed silently.



  • @tar said:

    PowerShell is actually pretty decent under the hood

    ...which makes it a real pity that it looks like this on the outside:



  • @RaceProUK said:

    Powershell is also far newer, and designed as a ground-up replacement for CMD.

    Right; and what's stopping the Linux community from replicating that? What's their excuse for having such an old, stupid, clearly wrong system still?

    @RaceProUK said:

    Saying it's better is a bit like saying a Ferrari is faster than a horse: true, but also an unfair comparison.

    Right; but it's 2015. WHY THE FUCK ARE THOSE IDIOTS RIDING THE HORSE INSTEAD OF TAKING THE FERRARI?



  • @Eldelshell said:

    For 30 years, using spaces in filenames has been a bad idea in *nix systems. That you don't know already to workaround those is another reason to stay away.

    So let me get this straight:

    Linux is great because people have to learn (by error, presumably) that spaces in filenames are trouble and avoid them? Even though it allows spaces in filenames. And if the user wants to use a space, well, then FUCK THEM!

    Compelling argument.



  • @asdf said:

    That's not the , the is that there are people who still use shell scripts in 2015.

    I beg to disagree; believe BOTH are a WTF.

    @asdf said:

    Also, NTFS isn't much better in this regard. It allows file names which no single tool on Windows will ever be able to do anything with.

    Sure, Windows has problems too. Who said otherwise?

    Not a tenth of the problems you have in Linux. And very few of the problems in Windows can lead to accidental data loss.



  • @Onyx said:

    The only way I see some progress being made if Debian, or RHEL, or some other big player made and / or adopted something like this and made it the default on their distro.

    Ok; but again: what's stopping them?

    @Onyx said:

    As I was writing this @asdf just mentioned systemd which is actually a fine demonstration of this point - no one gave a fuck about it until Debian adopted it. Slashdot exploded, accusations are being flung, but you know what? I use systemd right now. Sure, 99% of my services use the compatibility layer for systemv, and I still use service x restart instead of systemctl restart x, but it's there. Same could be done for bash. There would be kicking and screaming, but it would be doable.

    The reason people are bitching and whining on Slashdot is that systemd brings Linux close to parity with Windows, when it comes to boot and service management.

    Having the same features as Windows == becoming Windows == Satan Hitler Death Emergency Panic OMG


  • Winner of the 2016 Presidential Election

    @blakeyrat said:

    Sure, Windows has problems too. Who said otherwise?

    I wasn't disagreeing with you or trying to find excuses, I just took the opportunity to complain about a similar problem that has particularly annoyed me in the last few days.



  • @TwelveBaud said:

    The hell? What on earth could you possibly be doing that you hit it with any frequency at all?

    c:\Users\Swayde\Documents\WebSites\ThisIsAWebsiteIAmStoringOnMyLocalBoxBecauseImDoingSomeContractingWorkOnIt\TheHomePageOfTheWebsiteIsActuallyInThisFolderNotTheParentLikeYoudExpectSorry\ThisFolderContainsSomeOfTheErrorPagesButLookInTheOtherOneForThe500ErrorOk\TheseGuysMadeASeperateErrorPageForEverySinglePossible400ErrorTheyAreDedicatedManThatWasALotOfWork



  • @blakeyrat said:

    Ok; but again: what's stopping them?

    People who are steering the ship don't think anything's wrong. They grew up with warts and learned to live with them, even love them.

    Also, stuff like powershell is great when it works, but same as GUI, adds overhead and complexity. History has shown people will overlook the flaws for the sake of simplicity and lower barrier to entry. Case in point: the continuous success of PHP and javascript over competition.



  • I've seen people hit the MAX_PATH under a naming scheme which seemed to involve spamming the same few name components over and over:

    d:\Work\CompanyName\DivisionName\Services\Mainline\SourceCode\CompanyName\DivisionName\ProjectName\Imports\CompanyName\DivisionName\ImportedProjectName\SourceCode\CompanyName\DivisionName\ProjectName\FinallyAClassName.cs
    

    and them maybe you'd have the same file in a branch:

    d:\Work\CompanyName\DivisionName\Services\Branches\BranchName\SomeBranchClassifier\SourceCode\CompanyName\DivisionName\ProjectName\Imports\CompanyName\DivisionName\ImportedProjectName\SourceCode\CompanyName\DivisionName\ProjectName\FinallyAClassName.cs
    

    And then there's always someone who wants to go against the company "best practice" of putting all your work in d:\Work:

    c:\Users\PersonName\MyWork\CompanyName\DivisionName\Services\Branches\BranchName\SomeBranchClassifier\SourceCode\CompanyName\DivisionName\ProjectName\Imports\CompanyName\DivisionName\ImportedProjectName\SourceCode\CompanyName\DivisionName\ProjectName\FinallyAClassName.cs
    

    Of course the question "Why are you doing this? Why? WHY?!?!?" always went unanswered....



  • @flabdablet said:

    ...which makes it a real pity that [PowerShell] looks like this on the outside:

    It actually kinda reminds me of Perl...


  • Discourse touched me in a no-no place

    @TwelveBaud said:

    It is, since .NET runs on top of the Win32 subsystem, which for compatibility constraints can't ever increase MAX_PATH. If they made a .NET NT, though, then it wouldn't be.

    There's a, well… magic incantation you can put at the start of a filename to make it use the route without the filename size limit. (I have to look it up every time; it's not obvious.) When they're running on a capable platform (i.e., pretty much all the time; this is 2015, everyone!) the runtime could just add that at the front in the correct way and route around the problem automatically.

    I don't know why they don't. (I have my suspicions, but it's Easter so let's not cast too many aspersions… 😈 )



  • @cartman82 said:

    People who are steering the ship don't think anything's wrong.

    A.k.a. "it's always been done that way." The absolute worst reason for anything.



  • @tar said:

    And then there's always someone who wants to go against the company "best practice" of putting all your work in d:\Work:

    Your company best practice is a shit.

    @tar said:

    Of course the question "Why are you doing this? Why? WHY?!?!?" always went unanswered....

    Your alternative example is also a shit.

    You are double-shitting here. Your company is full of people doing dumb and wrong things, then telling other people the dumb and wrong things are the "best practice" and I hate you.


Log in to reply