Filesystem says what?



  • I closed a ticket today wherein I stated the following:


    "The system had a corrupted directory entry in /usr/somevendor/clip/stonefs7 where .ref-lock was a directory that was very much not a directory, except for the fact that it was a directory."

    The rationale...

    [code]
    me@twod01:/usr/somevendor/clip/stonefs7 > file .ref-lock

    .ref-lock: directory

    me@twod01:/usr/somevendor/clip/stonefs7 > cd .ref-lock

    .ref-lock: Not a directory.

    me@twod01:/usr/somevendor/clip/stonefs7 > cat .ref-lock

    cat: .ref-lock: Invalid argument

    [/code]

    I'm still not sure how the filesystem allows something to be in this particular state, or how to reproduce it. I certainly declared the name of this site loudly and robustly when I realised this was happening...



  • @wrosecrans said:

    I'm still not sure how the filesystem allows something to be in this particular state, or how to reproduce it.

    Corrupted metadata. Was the filesystem Reiser or one of the exts?



  • @morbiuswilters said:

    @wrosecrans said:
    I'm still not sure how the filesystem allows something to be in this particular state, or how to reproduce it.

    Corrupted metadata. Was the filesystem Reiser or one of the exts?

    I'm guessing it was stonefs.

     



  • @DaveK said:

    @morbiuswilters said:

    @wrosecrans said:
    I'm still not sure how the filesystem allows something to be in this particular state, or how to reproduce it.

    Corrupted metadata. Was the filesystem Reiser or one of the exts?

    I'm guessing it was stonefs.

    Oh. What the hell is stonefs?



  • @morbiuswilters said:

    @DaveK said:

    @morbiuswilters said:

    @wrosecrans said:
    I'm still not sure how the filesystem allows something to be in this particular state, or how to reproduce it.

    Corrupted metadata. Was the filesystem Reiser or one of the exts?

    I'm guessing it was stonefs.

    Oh. What the hell is stonefs?

    @wrosecrans said:

    "The system had a corrupted directory entry in /usr/somevendor/clip/stonefs7

    This post doesn't answer your question at all, but I just wanted to point that out.



  • @Ben L. said:

    This post doesn't answer your question at all, but I just wanted to point that out.

    Useful, as always.



  • I've seen NTFS do that; didn't know *n*x filesystems had the same feature.



  • @morbiuswilters said:

    @DaveK said:

    @morbiuswilters said:

    @wrosecrans said:
    I'm still not sure how the filesystem allows something to be in this particular state, or how to reproduce it.

    Corrupted metadata. Was the filesystem Reiser or one of the exts?

    I'm guessing it was stonefs.

    Oh. What the hell is stonefs?

    Awww, is five seconds googling too much like hard work?  There, there.  Let me help you out: http://www.google.com/search?q=stonefs+autodesk

    There's an interesting-looking result on the first page that says "WARNING Do not use the word “stonefs” as the name for your mount point directory. “Stonefs” is a reserved word, and can cause issues if used as the mount point directory name."  I wonder if that's relevant here.  If so, it may well be TRWTF.




  • @DaveK said:

    Awww, is five seconds googling too much like hard work?

    You know goddamn well it is.

    @DaveK said:

    Let me help you out: http://www.google.com/search?q=stonefs+autodesk

    Oh, autodesk, of course! Why didn't I think to Google for that.

    @DaveK said:

    There's an interesting-looking result on the first page that says "WARNING Do not use the word “stonefs” as the name for your mount point directory. “Stonefs” is a reserved word, and can cause issues if used as the mount point directory name."  I wonder if that's relevant here.  If so, it may well be TRWTF.

    Oh.


    What the hell is autodesk and why does it have its own filesystem?



  • @morbiuswilters said:

    What the hell is autodesk and why does it have its own filesystem?

    It's like ms-paint but with more shapes so people can draw buildings or gears (it also has a beautiful macro language). As for the filesystem:



    @RTFM said:

    Unlike the Stone filesystem, standard filesystems are openly accessible and standard filesystem tools or
    commands (outside of the Visual Effects and Finishing applications) can affect them. It is the responsibility
    of the system administrator to implement and enforce access rights in order to prevent media loss or
    performance degradation.

    (more exciting stuff here)



    I think on my next project I'll write my own filesystem for the same reasons.



  • So TRWTF is that a company implemented its own (apparently buggy) filesystem because normal filesystems are deemed to insecure?...



  • @dtech said:

    So TRWTF is that a company implemented its own (apparently buggy) filesystem because normal filesystems are deemed to insecure?...

    From scrolling through their manual they also define their own network protocols for file sharing and their own RAID implementation. Did someone just need to get rid of as much of their budget as possible before the end of the business year or something?



  • @morbiuswilters said:

    @DaveK said:
    Awww, is five seconds googling too much like hard work?

    You know goddamn well it is.

    Oh, if only you could be consistent, and also regard five seconds posting a reply as too much like hard work.

    @morbiuswilters said:

    @DaveK said:
    Let me help you out: http://www.google.com/search?q=stonefs+autodesk

    Oh, autodesk, of course! Why didn't I think to Google for that.

    Because you didn't bother googling "stonefs" by itself and see Autodesk mentioned in the first three results and apply the most minimal level of search-fu to think "I know, I'll add that to the search term to get better results".

    @morbiuswilters said:

    @DaveK said:
    There's an interesting-looking result on the first page that says "WARNING Do not use the word “stonefs” as the name for your mount point directory. “Stonefs” is a reserved word, and can cause issues if used as the mount point directory name."  I wonder if that's relevant here.  If so, it may well be TRWTF.

    Oh.


    What the hell is autodesk and why does it have its own filesystem?

    You've never heard of Autodesk?  They're only a huge multinational software corporation that basically owns the CAD, CGI and post-production market.  I refer you to the afore-mentioned GIYFS.

    I admit however that their own internal deranged motivation for creating their own FS is probably not googleable.

     



  • @RTFM said:

    Unlike the Stone filesystem, standard filesystems are openly accessible and standard filesystem tools or
    commands (outside of the Visual Effects and Finishing applications) can affect them. It is the responsibility
    of the system administrator to implement and enforce access rights in order to prevent media loss or
    performance degradation.

    Wow, so they made up their own filesystem specifically because they wanted a proprietary, non-standard filesystem that wouldn't be accessible from other applications - and they even have the nerve to call it a feature? That's... well amazing is the only word I can think of really.



  • @DaveK said:

    They're only a huge multinational software corporation that basically owns the CAD, CGI and post-production market.
     

    the fuck?

    Why would they own both 3DS and Maya and Softimage? That's a huge conflict of interest.



  • Monopoly courts only intervene when someone complains to a Congressman.

    How the fuck is it ok that Publicis owns both Razorfish and Digitas? ... well nobody complained.



  • @dhromed said:

    @DaveK said:

    They're only a huge multinational software corporation that basically owns the CAD, CGI and post-production market.
     

    the fuck?

    Why would they own both 3DS and Maya and Softimage? That's a huge conflict of interest.

    Because SGI sold off Alias Wavefront when they started going broke in the early naughties, and the Canucks they sold it to turned around and sold it to Autodesk, with little or no regulatory oversight.

     



  • @morbiuswilters said:

    What the hell is autodesk and why does it have its own filesystem?


    Autodesk is indeed the "somevendor" in the lightly anonymized original post.  However, I can't actually conclusively blame this particular issue on them, which is why I lightly anonymized, rather than trying to "name and shame."  The actual on disk filesystem where the problem happened was XFS, so I consider this WTF to be a RHEL / Linux kernel XFS driver bug rather than directly an Autodesk issue.

    Autodesk has a product called Flame (with a zillion slight variations over the years with clever names like Smoke, Fire, Inferno, Backdraft...) that dates back to the "good old days" of Irix on SGI systems.  It's a real time visual effects product that does real time playback off of a framestore.  20 years ago when the product was first created, it was impossible to reliably do the stuff they needed to do off of a standard UNIX filesystem in real time, so they had a product called "stone" which was a proprietary storage platform that let you play back multiple streams of uncompressed video in 1994.  (Which was very cool at the time!)  Over time, their product evolved as the platforms evolved around it, and people stopped using actual stonefs for storing frames and migrated to STANDARDFS, which is to say just storing files on a  normal file system, with some database files keeping track of the clip data that used to be natively part of stonefs.  In practice, it is almost always done on XFS due to the SGI legacy.  The name stonefs persists in the application in several spots for legacy reasons even on systems where actual stonefs is no longer used.

    Not to say that there aren't tons of WTF's in Flame and other Autodesk products.  Just that this one isn't their fault as far as I can tell.



  • I once had a directory that Windows Media Player created called " (Disc". Haunted my music folder for months before I figured out how to get rid of it (linux live cd). Windows absolutely could not figure the thing out.



  • @Master Chief said:

    I once had a directory that Windows Media Player created called " (Disc". Haunted my music folder for months before I figured out how to get rid of it (linux live cd). Windows absolutely could not figure the thing out.

    Windows: the OS for creating content. Deleting? What's that?



  • @wrosecrans said:

    The actual on disk filesystem where the problem happened was XFS, so I consider this WTF to be a RHEL / Linux kernel XFS driver bug rather than directly an Autodesk issue.

    That's odd.. XFS is one of the oldest and most mature file systems in Linux. My new guess is that it was some hardware problem.. controller or disk or something.

    @wrosecrans said:

    Autodesk has a product called Flame (with a zillion slight variations over the years with clever names like Smoke, Fire, Inferno, Backdraft...) that dates back to the "good old days" of Irix on SGI systems.  It's a real time visual effects product that does real time playback off of a framestore.  20 years ago when the product was first created, it was impossible to reliably do the stuff they needed to do off of a standard UNIX filesystem in real time, so they had a product called "stone" which was a proprietary storage platform that let you play back multiple streams of uncompressed video in 1994.  (Which was very cool at the time!)

    Well that makes sense. Thanks for the info!



  • @wrosecrans said:

    The actual on disk filesystem where the problem happened was XFS, so I consider this WTF to be a RHEL / Linux kernel XFS driver bug

    XFS on RHEL? 2.6.3x kernel? With XFS? Yeah, there's yer problem.



  • @morbiuswilters said:

    That's odd.. XFS is one of the oldest and most mature file systems in Linux. My new guess is that it was some hardware problem.. controller or disk or something.

    No, there is a known problem with XFS corruption on 2.6.3x kernels (I can't remember the exact range of kernel versions). Or he could have had some dumb combination of XFS options like nomembarrier and a RAID write cache.



  • @DaveK said:

    Oh, if only you could be consistent, and also regard five seconds posting a reply as too much like hard work.

    Replies are fun, Googling is not.

    @DaveK said:

    Because you didn't bother googling "stonefs" by itself...

    I actually did, but when I didn't immediately see a Wikipedia article I figured it was a waste of my time and if I just whined in the thread somebody would do my research for me. (Although it's really not my research, is it, since it's not my WTF and I don't actually give a fuck.)

    @DaveK said:

    You've never heard of Autodesk?

    No. Is it some kind of.. automated desk? Or.. like.. a desk car?

    @DaveK said:

    They're only a huge multinational software corporation that basically owns the CAD, CGI and post-production market.

    Oh. I was forced to use AutoCAD once in high school. My partner was in the grade ahead of me. He was a long-haired stoner who would come in baked every morning (first class of the day) and would put his head on the table and sleep. So while he was gently snoring on my mousing hand, I would draw out elaborate mazes and pretend that I was being asked to use a computer for something worthwhile--playing a video game about a labyrinthine space station--instead of something fucking pointless and boring--like drawing a goddamn house.

    Somehow I got a pretty good grade in that class. Another module in the class was working with computers. There was a sort of deconstructed motherboard, power supply and disk drives under acrylic. Somebody had long ago fucked up the Windows 95 login, so nobody could actually do anything. I knew how to boot with the help of the Win95 CD and then installed Half-Life and spent all day playing it. The computer was very shitty (200mhz, 64 megs of ram) but Half-Life ran okay enough and it was certainly a better use of taxpayer money than whatever the curriculum had planned.



  • @Vanders said:

    No, there is a known problem with XFS corruption on 2.6.3x kernels (I can't remember the exact range of kernel versions).

    Really? I've never encountered it, so I'm not sure I believe you. Also, that would have to be a really interesting fuck-up since XFS hasn't changed in a decade..



  • @morbiuswilters said:

    @Vanders said:
    No, there is a known problem with XFS corruption on 2.6.3x kernels (I can't remember the exact range of kernel versions).



    Really? I've never encountered it, so I'm not sure I believe you. Also, that would have to be a really interesting fuck-up since XFS hasn't changed in a decade..

    Oh I've never encountered it either, but it exists. And yes, the XFS on-disk format may not have changed, but the actual driver changes quite often, not to mention the underlying code like the block cache & VFS.



    Oh there you go, 2.6.17 apparently: http://xfs.org/index.php/XFS_FAQ#Q:_What_is_the_issue_with_directory_corruption_in_Linux_2.6.17.3F



    The problem being RHEL is so fucking stagnant that RHEL 5.x still runs 2.6.17 kernels.



  • @Ben L. said:

    Deleting? What's that?

    It's like unfriending. On a daily basis it feels like the unfriended entity is gone but they always show up in court to hurt you.



  • @Ronald said:

    @Ben L. said:
    Deleting? What's that?

    It's like unfriending. On a daily basis it feels like the unfriended entity is gone but they always show up in court to hurt you.

    Ben might be more familiar with the other side of unfriending: it's like how you stop seeing updates from someone, but you don't think they've died because everyone else you know can't stop talking about the mysteriously-vanished person and all of the exciting developments in his or her life.



  • @unfriending said:

    Ben might be more familiar with the other side of morbiuswilters: it's like how you stop seeing updatesposts from someone, but you don't think they've died because everyone else you know can't stop talking about the mysteriously-vanished person and all of the exciting developments in his or her life.



  • @Ben L. said:

    @unfriending said:
    Ben might be more familiar with the other side of morbiuswilters: it's like how you stop seeing updatesposts from someone, but you don't think they've died because everyone else you know can't stop talking about the mysteriously-vanished person and all of the exciting developments in his or her life.

    This is weaker than IRC trout slapping



  • To manipulate a file with a leading space in any component of the name (or that otherwise violates certain naming rules, including the 260-character limit), you must utter the magical incantation \\?\a:\bso\lute\path\to\ file to the Unicode version of an appropriate filesystem API. If the file is on a network share, use \\?\UNC\server\share\path\to\ file.

    Note that the Shell (explorer.exe, et al.) does not do this.



  • @Ronald said:

    @Ben L. said:
    @unfriending said:
    Ben might be more familiar with the other side of morbiuswilters: it's like how you stop seeing updatesposts from someone, but you don't think they've died because everyone else you know can't stop talking about the mysteriously-vanished person and all of the exciting developments in his or her life.

    This is weaker than IRC trout slapping

    It was a long 2 years. You weren't even born yet.



  • @electronerd said:

    To manipulate a file with a leading space in any component of the name (or that otherwise violates certain naming rules, including the 260-character limit), you must utter the magical incantation \?\a:\bso\lute\path\to\ file to the Unicode version of an appropriate filesystem API. If the file is on a network share, use \?\UNC\server\share\path\to\ file.

    Note that the Shell (explorer.exe, et al.) does not do this.

    The magic incantation is good to know about, but I regularly create shortcuts (.lnk files) with a leading space so that they'll sort just below the folders near the top of Explorer windows, and I've never needed to do anything special to manipulate them (XP, Server 2003, 7, Server 2012).

    I do occasionally run into the pathname length limitation, usually because somebody has saved a nameless Word document on a mapped drive and Word has used a huge slab of the text or the filename. I've always just created temporary drive letter mappings or used SUBST to deal with those, but I'll give the \\?\ thing a try next time.



  • @flabdablet said:

    I regularly create shortcuts (.lnk files) with a leading space

    Really? I thought leading spaces were one of the disallowed things. Trailing spaces definitely are. Oh well, I mostly use the \\?\ incantation for long file names.

    @flabdablet said:

    I do occasionally run into the pathname length limitation, usually because somebody has saved a nameless Word document on a mapped drive and Word has used a huge slab of the text or the filename. I've always just created temporary drive letter mappings or used SUBST to deal with those, but I'll give the \?\ thing a try next time.

    I have a feeling Word will not accept the \\?\ incantation, unfortunately. The Open dialog certainly won't accept it (since it's using the Shell). Another trick that might work for that is using the 8dot3 names of the file or some of the folders in the path, if generating them hasn't been turned off. (dir /x will show you them)

    One really unfortunate thing that doesn't support it: .NET. I'm getting really tired of needing to re-implement half of System.IO (okay, mostly just enough to enumerate/open/delete files) with P/Invoke every time I need to be able to handle a long path (which does come up in what I work on). .NET won't even let you use the 8dot3 trick, as it normalizes paths before operating on them.



  • @electronerd said:

    I'm getting really tired of needing to re-implement half of System.IO

    Do you work for autodesk



  • @Ronald said:

    @electronerd said:
    I'm getting really tired of needing to re-implement half of System.IO

    Do you work for autodesk

    Haha, no. But my options are (without abandoning C#): 1. Find a library that already takes care of this, evaluate it, get it cleared with management and Legal, get it purchased if necessary (not necessarily in that order), and use it in the two or three places where I need it, or 2. P/Invoke to CreateFile, FindFirst/NextFile, and DeleteFile, write a couple of very thin wrappers around them for convenience, and get on with my day. I chose #2. If I had to actually implement reading and writing as well, #1 might have been more attractive, but luckily FileStream has a constructor that takes SafeFileHandle, so all the subtle nuances of file I/O are still handled by the built-in code.



  • @electronerd said:

    To manipulate a file with a leading space in any component of the name (or that otherwise violates certain naming rules, including the 260-character limit), you must utter the magical incantation \?\a:\bso\lute\path\to\ file to the Unicode version of an appropriate filesystem API. If the file is on a network share, use \?\UNC\server\share\path\to\ file.
    You don't actually need \?\ prefix to manipulate filenames that start with a space - just don't use Explorer, because only it doesn't like leading spaces in filenames.



  • @ender said:

    just don't use Explorer, because only it doesn't like leading spaces in filenames.

    Under what conditions have you seen it fail? I can copy, rename and delete my space-led .lnk files just fine.



  • @flabdablet said:

    Under what conditions have you seen it fail? I can copy, rename and delete my space-led .lnk files just fine.
    Try renaming a file (not a link) to have a leading space in Explorer and observe what happens.



  • @electronerd said:

    To manipulate a file with a leading space in any component of the name (or that otherwise violates certain naming rules, including the 260-character limit), you must utter the magical incantation \?\a:\bso\lute\path\to\ file to the Unicode version of an appropriate filesystem API. If the file is on a network share, use \?\UNC\server\share\path\to\ file.

    Note that the Shell (explorer.exe, et al.) does not do this.

    Yup, I had to do that. Unfortunately I had to hunt that down myself, which was notably more difficult. But yeah.

    Interestingly, the shell commands themselves are not shielded from the leading space. Without that little \?, they fail in the same way as the explorer counterparts. Kind of WTFy that all the filesystem oversight takes place in the shell, and not in the filesystem itself.



  • @Master Chief said:

    Kind of WTFy that all the filesystem oversight takes place in the shell, and not in the filesystem itself.

    Actually, there are naming rules in at least four different places: the filesystem (though I think it mostly just stores whatever sequence of arbitrary 16-bit values it’s given for the name), the NT kernel object namespace manager, the Win32 subsystem, and the Shell. The \?\ escape basically tells the Win32 subsystem not to normalize or otherwise mess with the path before handing it off to the kernel (sort of). I believe one of the reasons the Shell doesn't support this is that there's a lot of code out there (including in the Shell APIs) that looks like TCHAR szPath[MAX_PATH]; (and remember, paths prefixed with \?\ can be longer than MAX_PATH).



  • @morbiuswilters said:

    Replies are fun, Googling is not.

    This right here explains a lot about morbius. When you wonder why he posts what he does just remember, he's looking for fun replies.
    Almost consider him a troll with this definition, but he is more fun so I give him a free pass. Not that free passes from me really matter.



  • @ender said:

    Try renaming a file (not a link) to have a leading space in Explorer and observe what happens.

    I see Explorer trim the leading space(s). On the other hand, deleting a file named with a leading space, or copying and pasting or dragging and dropping one such into a folder that is itself named with a leading space, works fine; so I'm not convinced this is Explorer being unable to deal with leading spaces at the filesystem API level. It smells more like "helpful" UI design to me.

    It's a pity the Welcome screen isn't similarly helpful. More than once I've seen a staff member unable to log into their Windows account, desperately typing and re-typing multiple variants of their password with and without Caps Lock while failing to notice the leading space they'd inadvertently entered in the Username box.



  • @Master Chief said:

    Interestingly, the shell commands themselves are not shielded from the leading space. Without that little \?, they fail in the same way as the explorer counterparts.

    Steps to reproduce? I've never seen cmd.exe have trouble with leading spaces; in fact it's exactly what I used to make the files I was just then testing in Explorer.



  • @KattMan said:

    When you wonder why he posts what he does just remember, he's looking for fun replies.
    Almost consider him a troll with this definition, but he is more fun so I give him a free pass.

    I like to consider myself a comedic performance artist using the medium of creative writing and adopting trollish affectations.


  • ♿ (Parody)

    @morbiuswilters said:

    I like to consider myself a comedic performance artist using the medium of creative writing and adopting trollish affectations.

    I've always known you were a humorist. It can't be an accident that your avatar is usually smiling.



  • @morbiuswilters said:

    I like to consider myself think I'm a comedic funny performance artist cunt using the medium of creative writing lies and adopting trollish affectations style and hygiene tips from Richard Stallman trollish affectations.
    FTFY.



  • @morbiuswilters said:

    @DaveK said:
    Oh, if only you could be consistent, and also regard five seconds posting a reply as too much like hard work.

    Replies are fun

    Only for you, Morbs. Only for you.



  • @electronerd said:

    @flabdablet said:
    I regularly create shortcuts (.lnk files) with a leading space

    Really? I thought leading spaces were one of the disallowed things. Trailing spaces definitely are. Oh well, I mostly use the \\?\ incantation for long file names.

    Why can't I make a file that has spaces where I want them? If I want to make a file with three colons and a less than sign followed by two spaces, I SHOULD BE ABLE TO DO THAT.

    At least linux allows you to make a file named if you want to. That's like a basic human right.



  • @Ben L. said:

    @electronerd said:
    @flabdablet said:
    I regularly create shortcuts (.lnk files) with a leading space

    Really? I thought leading spaces were one of the disallowed things. Trailing spaces definitely are. Oh well, I mostly use the \\?\ incantation for long file names.

    Why can't I make a file that has spaces where I want them? If I want to make a file with three colons and a less than sign followed by two spaces, I SHOULD BE ABLE TO DO THAT.

    At least linux allows you to make a file named if you want to. That's like a basic human right.

    Linux is a bad example of file systems with sane policies concerning file names.


Log in to reply