Bringing down the main domain/file server



  • I saw the thread titled "NTFS and undeleteable files" and that reminded me of a story from my high school days, where I was the WTF and I managed to cost the school a lot of time and money.

    Back when I was in junior high, I figured out how to access the network storage of any arbitrary user (hidden shares with no ACLs FTW!). As a joke, I went into a friend's storage and made a text file saying "Follow the white rabbit" (this was shortly after the Matrix sequels were released). Then I made a folder named "White Rabbit" and inside that folder I put another "White Rabbit" folder and so on, until I hit the nesting limit of the file system. Then I put a shortcut back to the top-level "White Rabbit" folder. If my friend wasn't paying attention and didn't notice the icon change, he could have been clicking for hours! He found it later and for some reason Windows wouldn't let him delete it. We left it and forgot about it.

    Two or three years passed. Now we were in high school. I was a techie and I knew the sys admin pretty well by then. I spent the summers working under him doing random tech stuff around the school.

    One of those summers, he finally got around to doing maintenance on the main server. It ran Novell NetWare and functioned as a domain controller and primary file server for the entire school. Our tape backup server ran backup every night, and it would always crash out and require a hard reset when it tried to back up the file system of the main server. Apparently it had been doing this for years and he just never had time to investigate it.

    He decided it must be flaky hardware somewhere, so he replaced the main server and tape backup server a few years earlier than planned. The new servers were up and running and the original disks were cloned over to new ones but the problem didn't go away. Finally he called up the school's Novell contact and he came down to take a look. This guy worked out of Omaha, which was about an hour and a half away, and Novell charged the school several hundred dollars an hour from when he left Omaha until he returned. He spent three or four days at our school (thankfully he did have other stuff he needed to take care of, such as NetWare updates and other maintenance on most of the servers) but he found the problem--that "White Rabbit" thing was nested too deep and the NetWare operating system couldn't understand it. The backup would fail and NetWare wouldn't allow us to delete it (it's weird that Windows let me place it over the network share). I think in the end he took down the server, booted it from a Linux live CD, and was able to delete the folder from there. After that the tape backup process ran flawlessly.

    So I personally brought down my high school's tape backup every single night for two or three years solid! And cost them at least ten thousand dollars to get it diagnosed and corrected. I'm a WTF. This and a couple other minor incidents--such as accidentally cutting the school's T1 line with wire cutters (by the way, T1 lines carry enough voltage to give you a good shock as I discovered when I tried to fix it before anybody noticed)--should have put me on thin ice, though I never did get in any trouble for it.



  •  Awesome!



  • you are not allowed anywhere near my office....  

     



  •  +1 coolness



  • Something doesn't smell right about this story:

    • NetWare doesn't have hidden shares (NetWare doesn't even have shares!!).
    • NetWare doesn't have a path length limit (but it did have a maximum directory nesting depth of 100 in the early days), the limit is in the client.  If you run the backup software on the server, it can backup very deep directory structures (arbitrarily deep in newer versions).  Even stranger, since the limit is on the client side and NetWare support the concept of "map root", the path length for a specific file depends on how you get to it.
    • Mounting an NSS or NWFS volume from a Linux live CD is very unlikely.  GParted doesn't support it, and I doubt the more general purpose releases do either.  SUSE can mount NSS, but I doubt it could when you were back in junior high.  Novell added the feature sometme after they acquired SUSE in 2004.
    • If this happend before NetWare 6, then the Linux option wouldn't be available.  If this happened after NetWare 6, then the Linux option would be unnecessary as the console ran Java programs, including a file manager.  Besides, the simple solution is just to map a drive on a Windows client a few subdirectories up, then the paths become shorter and they can be manipulated.
    • NetWare can't be a domain controller.  Pre-1993, it simpy didn't support anything similar to Windows domains.  Post-1993, it would be an NDS Replica Server.  The name changed to eDirectory in later versions.

     



  • @Jaime said:

    Something doesn't smell right about this story:

    <details>

    So what you're saying is that he Remy'd his own story?



  • @Jaime said:

    Something doesn't smell right about this story:

    • NetWare doesn't have hidden shares (NetWare doesn't even have shares!!).
    • NetWare doesn't have a path length limit (but it did have a maximum directory nesting depth of 100 in the early days), the limit is in the client.  If you run the backup software on the server, it can backup very deep directory structures (arbitrarily deep in newer versions).  Even stranger, since the limit is on the client side and NetWare support the concept of "map root", the path length for a specific file depends on how you get to it.

    NetWare definitely supported shares. They didn't show up in Network Neighborhood but you could get to them with a direct path. I'm not sure about the path limit stuff, I embellished that part of the story slightly. I don't know why the folder thing caused it to crash so I made an assumption about nesting limits.
    @Jaime said:

    • Mounting an NSS or NWFS volume from a Linux live CD is very unlikely.  GParted doesn't support it, and I doubt the more general purpose releases do either.  SUSE can mount NSS, but I doubt it could when you were back in junior high.  Novell added the feature sometme after they acquired SUSE in 2004.
    • If this happend before NetWare 6, then the Linux option wouldn't be available.  If this happened after NetWare 6, then the Linux option would be unnecessary as the console ran Java programs, including a file manager.  Besides, the simple solution is just to map a drive on a Windows client a few subdirectories up, then the paths become shorter and they can be manipulated.
    • NetWare can't be a domain controller.  Pre-1993, it simpy didn't support anything similar to Windows domains.  Post-1993, it would be an NDS Replica Server.  The name changed to eDirectory in later versions.

     

    I'm a lot younger than you're assuming. The issue was fixed around late 2005 or early 2006. Also, domain controller may be the wrong terminology, but the eDirectory stuff sounds familiar. In any case, though it wasn't a Windows domain, it definitely handled logins. All PCs on the network had to have Novell client software installed. Without that and without a valid login, you couldn't access shares, GroupWise email, or even the Internet.



  • Nice story. I don't see how you're TRWTF though. Not only did you use the filesystem to play a clever prank on your friend (a laudible goal in itself), but you simultaneously outdid Novell's entire QA department (wait, they have one?). You, sir, should have a medal.



  • @Faxmachinen said:

    Nice story. I don't see how you're TRWTF though. Not only did you use the filesystem to play a clever prank on your friend (a laudible goal in itself), but you simultaneously outdid Novell's entire QA department (wait, they have one?). You, sir, should have a medal.

    The problem here isn't Novell.  It may be Microsoft (for limiting paths to 260 characters in the Windows API), or the backup software vendor, or the person who set the backup software up.  NetWare never had a problem with long path names, they have supported arbitrarily long paths for over ten years, and 100 directory levels deep, each with 255 character names, before that.


  • @mott555 said:

    @Jaime said:

    Something doesn't smell right about this story:

    • NetWare doesn't have hidden shares (NetWare doesn't even have shares!!).
    • NetWare doesn't have a path length limit (but it did have a maximum directory nesting depth of 100 in the early days), the limit is in the client.  If you run the backup software on the server, it can backup very deep directory structures (arbitrarily deep in newer versions).  Even stranger, since the limit is on the client side and NetWare support the concept of "map root", the path length for a specific file depends on how you get to it.


    NetWare definitely supported shares. They didn't show up in Network Neighborhood but you could get to them with a direct path. I'm not sure about the path limit stuff, I embellished that part of the story slightly. I don't know why the folder thing caused it to crash so I made an assumption about nesting limits.
    You were correct that you ran into a path length limit, but you were incorrect in your assessment that it had anything to do with NetWare.  Your story would have been 100% correct and still entertaining if you had removed all of the unnecessary Novell bashing from it.  Also, you weren't the WTF, the tech who took thousands of dollars to fix a common and trivial problem is.  Taking down a server to boot it from a Linux CD to access a long path is the work of someone who knows Linux, but has no idea how NetWare works.

    BTW, the word "share" has a pretty specific meaning in this context and when used as "hidden share", has an even more specific meaning that is incorrect when applied to NetWare.  The only thing on a NetWare server that would get confused with a share is a volume.  However, creating a volume for each user is extraodinarily impractical, leading to my statement that this isn't likely to have actually happened the way it was described.



    1. Open MS Word or Excel.  (Or any MS program with VBA support)
    2. Press Alt-F11 to open the VBA Editor
    3. Press Control-G to open the Immediate pane
    4. Type: mkdir "C:\test \"      (Note space before last backslash)
    5. Press enter.
    6. Try to delete the newly created folder.


  • @fourchan said:

    1. Open MS Word or Excel.  (Or any MS program with VBA support)
    2. Press Alt-F11 to open the VBA Editor
    3. Press Control-G to open the Immediate pane
    4. Type: mkdir "C:\test \"      (Note space before last backslash)
    5. Press enter.
    6. Try to delete the newly created folder.

     

    Very good.  I tried, without success, to delete or rename it in Explorer, CMD and using VB Script. Found a way to do it thanks to Google but don't think I could have done so on my own.



  • @fourchan said:

  • Type: mkdir "C:\test \"      (Note space before last backslash)
  •  

    Nice. Probably an oldie but works fined on windows 7.

    Interestingly, powershell doesn't see the folder, cmd.exe does.



  • @fourchan said:

    Type: mkdir "C:\test \"

    I once managed to create a file or folder (don't remember what, exactly) named just " " (yes, just a single space char). That one was hellishly difficult to get rid of...

    How did this come about? I can't remember the details exactly, but I think it was when I installed some piece of software, and the installer asked me for the target path, I mistyped and added a rogue space somewhere around the "C:\" part (I think I typed it as "C: \" by accident), and the installer took it and ran with it...

    One trick that I learned that day is to remember that Windows is still maintaining those old 8.3 short names behind the scenes, in parallel to the long names (it just hides them from view, but you can see them with a "dir /x", for example) and you can just use those instead to deal with files/folders that have problematic names.



  • @Anonymouse said:

    @fourchan said:

    Type: mkdir "C:\test \"

    One trick that I learned that day is to remember that Windows is still maintaining those old 8.3 short names behind the scenes, in parallel to the long names (it just hides them from view, but you can see them with a "dir /x", for example) and you can just use those instead to deal with files/folders that have problematic names.

     

    I don't know how this would work with anything Windows, but back in the DOS days I made a QBASIC program that could create files/directories with spaces in them. This is on MS-DOS 5.0 and 6.22. Some programs could work with those filenames but most couldn't!



  • @Anonymouse said:

    I once managed to create a file or folder (don't remember what, exactly) named just " " (yes, just a single space char). That one was hellishly difficult to get rid of...
    It's worse when you have a folder (or file) named CON, AUX, LPT1 etc. Luckily, FAR Manager (which is my preferred file manager) doesn't seem to have any problems with these (it also doesn't have the 260-character path limit, so it can easily get rid of the deeply nested directories).@Zemm said:
    I don't know how this would work with anything Windows, but back in the DOS days I made a QBASIC program that could create files/directories with spaces in them. This is on MS-DOS 5.0 and 6.22. Some programs could work with those filenames but most couldn't!
    Way back when we had a 286 at home, and my father bought a mouse, it came with a drawing program called Dr.Genius. It had no problems working with files with spaces in names - even though pretty much everything else barfed on those files. I remember reading somewhere that space was actually a perfectly legal character in filenames in DOS, but since spaces were normally used as a separator (and there was no escaping or quoting), most programs didn't support them.



  • I personally like this filename:

    [img]http://www.dynamicarcade.co.uk/Dumping%20Ground/wtf%20filename.png[/img]

    Took that screenshot myself a couple of years ago.



  • @fourchan said:

    1. Open MS Word or Excel.  (Or any MS program with VBA support)
    2. Press Alt-F11 to open the VBA Editor
    3. Press Control-G to open the Immediate pane
    4. Type: mkdir "C:\test \"      (Note space before last backslash)
    5. Press enter.
    6. Try to delete the newly created folder.

     

    ummm, 

    rmdir "C:\test \" -- still in excel of course

    Unix equivilant:
    $ echo > -i

    or if you are feeling mean

    $ echo > -rf

     



  • @Thief^ said:

    Took that screenshot myself a couple of years ago.
    You can (or at least could) do that with NTFS-3g. Don't remember if Windows let you delete such file though.



  • @Rick said:

    rmdir "C:\test \" -- still in excel of course

    1337 HaXXor sk1llz.

    Actually it's quite fun trying to solve it outside of Excel. MSDN has some help on it.



  •  I managed to nuke a machine with

     

    mkdir spam

    cd spam

    copy ..\spam.bat

    spam.bit

     

    (saved as spam.bat)

     

     the person whose machine it was turned it off mid-run, and it left the deepest folder pointing at the root of C for some reason. Being an old 286 with whatever version of DOS was around back them meant you couldn't delete a folder with anything in it, so when he did a del *.* on c:\spam\spam\<ad nauseum>\ he was thoroughly hacked off...

     



  • @Jaime said:

    You were correct that you ran into a path length limit, but you were incorrect in your assessment that it had anything to do with NetWare.  Your story would have been 100% correct and still entertaining if you had removed all of the unnecessary Novell bashing from it.  Also, you weren't the WTF, the tech who took thousands of dollars to fix a common and trivial problem is.  Taking down a server to boot it from a Linux CD to access a long path is the work of someone who knows Linux, but has no idea how NetWare works.

    BTW, the word "share" has a pretty specific meaning in this context and when used as "hidden share", has an even more specific meaning that is incorrect when applied to NetWare.  The only thing on a NetWare server that would get confused with a share is a volume.  However, creating a volume for each user is extraodinarily impractical, leading to my statement that this isn't likely to have actually happened the way it was described.

    How much does Novell pay you? Alternatively, who pissed in your cornflakes this morning?



  • @Zemm said:

    @Anonymouse said:

    @fourchan said:

    Type: mkdir "C:\test \"

    One trick that I learned that day is to remember that Windows is still maintaining those old 8.3 short names behind the scenes, in parallel to the long names (it just hides them from view, but you can see them with a "dir /x", for example) and you can just use those instead to deal with files/folders that have problematic names.

     

    I don't know how this would work with anything Windows, but back in the DOS days I made a QBASIC program that could create files/directories with spaces in them. This is on MS-DOS 5.0 and 6.22. Some programs could work with those filenames but most couldn't!

    You reminded me of back in the day writing a really basic DOS installer in C.  I stuffed up some null termination and rather than creating c:\thing\docs, c:\thing\bin etc, my recursive directory creator created a million directories a billion levels deep with all the crap it could find from the uninitialised and unprotected heap.  There were graphics, spaces, foriegn characters, chars <32.  You name it, it was there.  Thank god for Norton Disk Doctor!

    The other high-jinks was to use NDD to create a file called *.* on a colleagues drive and watch the fun when they try to delete it.

    Ah, happy days.



  • I just tried it and was able to delete it in cmd.exe with rmdir ".\test "



  • @Rick said:

    Unix equivilant:
    $ echo > -i

    or if you are feeling mean

    $ echo > -rf

    $ echo > -i
    $ echo > -rf
    $ ls
    -i  -rf
    $ rm -- -i -rf
    $ ls
    $ echo > -i
    $ echo > -rf
    $ ls
    -i  -rf
    $ rm ./-*
    $ ls
    $ 
    

    Those are trivial to get rid of (GUI file managers have zero issue with them). The "--" is a feature of gnuopt AFAIK as most of the GNU core-utils supports it (it means everything after is a file, not an option), the ./-* is a more portable means (using a glob).


    [Edit]: To pre-empt the inevitable "but only *NIX gurus would know about that"; a novice can use a GUI or CUI based file manager, they can move every other file out of the directory then delete the directory, or they can do: "rm mktemp -i -rf" which in many implementations works the same as the -- solution (since in those implementations they stop looking for argument flags once they see the first file on the argument list). That would only be an equivolent is most tools choked on it when those files were present in the explicitly named directory (eg: if they were in ~/test/ and rm -r ~/test/ failed).


Log in to reply