How to handle disk space problems



  • This week I was asked to look into why a log import job has been failing for sometime.  Apparently it hasn't logged any data from the import process since sometime last year. Oh those pesky logfiles.  What on earth do you do on a modern Linux operating system when those naughty logfiles get to large and they start using all the filesystem space?  Why you kill them off of course!!

    Every night at midnight our past junior admin knothead ran this script from cron:

    #!/bin/sh
    # This script clears the apache logs on /var.
    # That was apparently the problem causing the forums to error
    # out on Jun 21-22.
    /bin/cat /dev/null > /var/log/httpd-access.log

     

    GENIUS!



  • Is it that big of a deal?  I clear out logs nightly as well.  Of course, I don't really care about the data, but it seems like diagnosing this would be easy and if somebody needs the log data it can be bzipped and sent to the person. 



  •  But isn't that script just replacing a several-gigabyte log file with a several-gigabyte file of... nothing?



  • @Isuwen said:

    But isn't that script just replacing a several-gigabyte log file with a several-gigabyte file of... nothing?

    Not on GNU/Linux.  "cat /dev/null > file" and "cp /dev/null file" both truncate the file, although I usually see the cp idiom more often. 



  • @Isuwen said:

     But isn't that script just replacing a several-gigabyte log file with a several-gigabyte file of... nothing?

    You're thinking of /dev/zero. /dev/null produces end-of-file when you read from it.



  •  logrotate anyone?



  •  Feh... *I'D* have just bought one of those 1.5 TB Seagate hard drives to store the log files on when I ran out of room and never rebooted the computer.



  •  Is there any reason why he didn't use rm instead of catting to /dev/null ?



  • Hmmm... I've put something similar on a server we support, but for fetchmail's log. Their partitions, and /var in particular, were tiny and they were running out of space. It was either that or I'd have to start deleting their emails. Not to mention the server is ancient and on its way to be completely replaced so adding HDDs to the existing one didn't make sense.

    On the other hand of course it's not like last week's fetchmail logs are likely to help you with anything...

     



  • @morbiuswilters said:

    "cat /dev/null > file" and "cp /dev/null file" both truncate the file, although I usually see the cp idiom more often

    Because ": > /var/log/whatever" is too fast? 



  •  there's no need for ":". a simple "> /lib/ld-linux.so.2" is enough for the job.



  • @Lord Illidan said:

    Is there any reason why he didn't use rm instead of catting from /dev/null ?
    I'd say file permissions.



  • @Lord Illidan said:

     Is there any reason why he didn't use rm instead of catting to /dev/null ?

    If you delete the file, when it gets recreated it might not have the same owner, group or perms.  Truncating it instead guarantees that they stay the same (also preserves the file creation time, come to think of it, although that's probably just an incidental side-effect rather than a reason for doing it that way).  It's a standard UNIX-ism.



  •  @TehFreek said:

    logrotate anyone?

    Absolutely. 

    It's been a while since I had to set it up but I think Red Hat / Fedora might have even come with a script to rotate apache log files (I think it was included in the apache RPM).



  • @PhillS said:

     @TehFreek said:

    logrotate anyone?

    Absolutely. 

    It's been a while since I had to set it up but I think Red Hat / Fedora might have even come with a script to rotate apache log files (I think it was included in the apache RPM).

     

    My predecessor never figured out how log rotate worked. Here is his novel solution for dealing with the Apache logs.

     

    ErrorLog "|/usr/local/apache/bin/rotatelogs /usr/local/apache/logs/mycompany.%Y%m%d.error.log 86400"
    CustomLog "|/usr/local/apache/bin/rotatelogs /usr/local/apache/logs/mycompany.%Y%m%d.access.log 86400" combined env=!misc_file
    CustomLog "|/usr/local/apache/bin/rotatelogs /usr/local/apache/logs/mycompany.%Y%m%d.access.flv.log 86400" combined env=flv_file   

    CustomLog "|/usr/local/apache/bin/rotatelogs /usr/local/apache/logs/mycompany.%Y%m%d.access.pdf.log 86400" combined env=pdf_file

     He never deleted the log files either

     



  • @Lord Illidan said:

     Is there any reason why he didn't use rm instead of catting to /dev/null ?


    Yes. In Unix, inodes (in most contexts, files) are not deleted until there are no processes holding them open. Most log-generating processes keep the log file open forever, so in order to empty a log file, you have to restart the process (or send it a magic signal like SIGHUP which the process may or may not interpret as "close and reopen all my log files"). So if you rm an open log file, the process which opened that file will happily continue writing to it, taking up more and more space.

    If you have access to a Linux box, you can actually see this happening. Open a terminal and do this:

    % cat >somefile

    Then open another terminal and look at the cat process's file descriptors:

    % ls -l /proc/`pgrep cat`/fd
    total 0
    lrwx------ 1 schad schad 64 Jan 22 11:37 0 -> /dev/pts/0
    l-wx------ 1 schad schad 64 Jan 22 11:37 1 -> /home/schad/somefile
    lrwx------ 1 schad schad 64 Jan 22 11:37 2 -> /dev/pts/0

    Conveniently, Linux creates a quasi-symlink to the file that cat's got open. Now, in your second terminal, mv the file and do it again:

    % mv somefile someotherfile
    % ls -l /proc/`pgrep cat`/fd
    total 0
    lrwx------ 1 schad schad 64 Jan 22 11:37 0 -> /dev/pts/0
    l-wx------ 1 schad schad 64 Jan 22 11:37 1 -> /home/schad/someotherfile
    lrwx------ 1 schad schad 64 Jan 22 11:37 2 -> /dev/pts/0

    So the file cat has open has followed the mv (because mv, in this circumstance, isn't moving the file at all, it's just renaming it; if you were moving to a different filesystem, then the file would be copied to the new destination and the original deleted). Now you can delete the file to see what happens:

    % rm someotherfile
    % ls -l /proc/`pgrep cat`/fd
    total 0
    lrwx------ 1 schad schad 64 Jan 22 11:37 0 -> /dev/pts/0
    l-wx------ 1 schad schad 64 Jan 22 11:37 1 -> /home/schad/someotherfile (deleted)
    lrwx------ 1 schad schad 64 Jan 22 11:37 2 -> /dev/pts/0

    So now Linux is telling you that the file in question has been deleted. You can create a file with the same name, but if you send some data for cat to write to it, it'll get written to the deleted file, not the newly-created one.

    If you want, you can make the effect really clear by copying a large (1GB+) file to something else, then, mid-copy, deleting it. The copy will continue along just fine (though you may get errors about not being able to set the permissions or ownership at the end), and if you watch df, you'll see that it continues to consume disk space. At the end of the copy, all the disk space will be immediately released.

    This all makes perfect sense if you understand the way Unix does filesystems. The fact that this behavior flies in the face of what any sensible person would expect to happen is one of the reasons that I'm not a Unix zealot, even though it is probably my favorite family of modern operating systems.



  •  Thanks all for your explanations!



  • @schad said:

    This all makes perfect sense if you understand the way Unix does filesystems. The fact that this behavior flies in the face of what any sensible person would expect to happen is one of the reasons that I'm not a Unix zealot, even though it is probably my favorite family of modern operating systems.

    Actually, it has its uses.  Not being able to replace an in-use file (e.g. executables such as the kernel or system DLLs) is one of the main reasons why windoze needs rebooting so often when you update or install anything.



  • @DaveK said:

    Actually, it has its uses.  Not being able to replace an in-use file (e.g. executables such as the kernel or system DLLs) is one of the main reasons why windoze needs rebooting so often when you update or install anything.
    You can rename an in-use file, though, and put another file under the original name. Windoze actually needs rebooting "so often" because a) installer writers are idiots, and b) while Linux library authors strive to make sure libraries are both forward- and backward-compatible, and have extensive package management (e.g. Yum, APT, RPM) systems with install and uninstall hooks to make sure everything Just Works, Windows is more wild-westy and has no centralized package manager (Windows Installer doesn't count because though it's as powerful as Yum/APT/RPM it is nowhere near as authoritative) so the only way to ensure that all versions of a DLL are compatible with each other is to make sure there's only one version.

    And then there's WinSxS...



  • @DaveK said:

    Not being able to replace an in-use file (e.g. executables such as the kernel or system DLLs)

    Regarding kernel DLLs (drivers), though, you can replace them all day long. Kernel DLL/driver file is never "in use"/locked. The new copy will be loaded as soon as you stop all devices that are using it, and start the device again. That worked since Windows NT 3.1. I remember that many drivers in Linux used to be/are in the same binary as kernel. Loadable kernel modules in Linux are pretty recent development (compared to Windows support for that feature).



  • @TwelveBaud said:

    You can rename an in-use file, though, and put another file under the original name.
    Bwahahaha.  Theoretically, maybe.  In practice, Windows' obiquitous methods to disallow this are almost never [i]not[/i] used.




  • @alegr said:

    Regarding kernel DLLs (drivers), though, you can replace them all day long. Kernel DLL/driver file is never "in use"/locked. The new copy will be loaded as soon as you stop all devices that are using it, and start the device again. That worked since Windows NT 3.1. I remember that many drivers in Linux used to be/are in the same binary as kernel. Loadable kernel modules in Linux are pretty recent development (compared to Windows support for that feature).
    Linux has supported it for [i][b]fifteen years[/b][/i], that is not "pretty recent".  Of course Windows had it first, windows was there first.



  •  @alegr said:

    [snip]
    Oooh, ooh! I want your copy of Windows where loaded modules are never "in use"! I want the one where if you replace a DLL the old version will continue to be used, even for new "devices", until all handles to the old version are released! Where do I sign!??

    </sarcasm>

    Windows' loader locks loaded modules against write and delete, though not rename. You cannot copy a DLL over a DLL that a program has loaded. You can rename an in-use DLL, and put a new one in its place, in which case running programs will continue using the old copy and newly started programs will use the new copy, which if the two copies use shared memory or other IPC could cause all applications using both to crash, since neither version will understand the other's messages; otherwise, everything just continues working since there's nothing to be confused about.

    I'm sorry for being rude, but this is simple stuff you can prove just by trying to delete shell32.dll. It doesn't work. Hasn't on any NT since before NT Advanced Server 3.51.



  • @Corona688 said:

    @TwelveBaud said:
    You can rename an in-use file, though, and put another file under the original name.
    Bwahahaha.  Theoretically, maybe.  In practice, Windows' obiquitous methods to disallow this are almost never not used.
    Thanks for the double negative.
    You can't write to or delete any in-use DLL because the module loader always opens DLLs as shared for read only. You can always rename an in-use DLL unless NTFS security prevents you. (Which, up through Windows Fundamentals, only realistically happens when you're not running as an administrator; most applications don't set up access control on their files, so they inherit the default "Admins:Full Control, Creator:Full Control, System:Full Control, Users:Read" permission set.)



  • @Corona688 said:

    Linux has supported it for fifteen years, that is not "pretty recent".  Of course Windows had it first, windows was there first.
    Well, if Linux has had it for fifteen years, it's beating Windows by about six. Under Windows NT 4.0, executive-level drivers were baked into the Hardware Abstraction Layer, which needed to be recompiled or patched for each additional driver, according to my research. Windows 2000 added the ability to run these as services.



  • @TwelveBaud said:

    Windows' loader locks loaded modules against write and delete, though not rename. You cannot copy a DLL over a DLL that a program has loaded

    For reading comprehension training purposes, note the keyword "kernel" in my posting. Kernel modules (all those pesky .sys files, and some of them are named as DLL) are not backed by their file. They are loaded to kernel space, and the file handle is closed. The file after that is not locked anymore. You can copy an updated file without any problems. This is as opposed to user mode modules, which are backed by a file mapping object associated with their file.



  • @TwelveBaud said:

    @Corona688 said:

    Linux has supported it for fifteen years, that is not "pretty recent".  Of course Windows had it first, windows was there first.
    Well, if Linux has had it for fifteen years, it's beating Windows by about six. Under Windows NT 4.0, executive-level drivers were baked into the Hardware Abstraction Layer, which needed to be recompiled or patched for each additional driver, according to my research. Windows 2000 added the ability to run these as services.

    And here you're wrong, too.

    Windows 2000 added WDM subsystem for plug-n-play (which was pretty major feature, by the way). Now the devices were enumerated by bus drivers and could be easily installed/loaded/unloaded along with the actual physical devices. The drivers were loaded by demand. NT 4.0 had rudimentary PNP for PCI devices. There (and in NT 3+), if you wanted to update a driver on the fly, you needed to stop its "service" by using (very "appropriately") one of the options of "net" command, and start it again. The image would be reloaded then. The drivers were loaded at system boot.

    The drivers have always been separate from HAL and Executive+IO. By the way, HAL is pretty small module which controls hardware-specific things. It's a separate binary from NT kernel (executive).

     



  • @Corona688 said:

    Linux has supported it for fifteen years, that is not "pretty recent".  Of course Windows had it first, windows was there first.

    I thought modprobe/modutils was first available for kernel 2.2 circa 1999.



  • @alegr said:

    I thought modprobe/modutils was first available for kernel 2.2 circa 1999.
    modprobe, yes. insmod has been around since kernel 2.0 (1996) and perhaps earlier.


Log in to reply