I love mondays



  • You know the week's gonna be fun when your admin tells you on monday morning:
    "btw, i've set our server to reboot every 3 hours, and it seams this increases its uptime."

    In such moments, you regret you're talking via text chat because the other side cannot hear
    the screams of pain that follow such statements.


    In addition, he happens to be the admin of the company we both work in and as he was going in holiday some weeks, I end up being assigned to monitor his emails in order to check no customer mail was lost.

    When we finally managed to get him give us the right password, further shards of pain followed as I was downloading about 3000 emails from his account. Now, you could think that most of it is spam, but there were barely 20 spams, 95% of the rest were mails from various cron jobs he had setup on the server.

    For example, about 2000 were from a script sending a mail every minute telling how much space left there is on the server (we use about 800Mb max while there is 3Gb available...), or another sending the list every minute (it seems but I cannot verify as I don't know the code of the script) as well of people who downloaded our softwares, which is not bad in itself, but it's an incremental list so the first mail of the day had one person while the last one of the day had all the names, so that our senior marketing person can spend hours to copy/paste the names in an excel spreadsheet, putting each data in the right column (the words "Use a smegging DB !" must be unknown around here).


    Captcha: brillant



  • [quote user="WIldpeaks"]

    "btw, i've set our server to reboot every 3 hours, and it seams this increases its uptime."

    For example, about 2000 were from a script sending a mail every minute telling how much space left there is on the server [/quote]
    <font face="tahoma,arial,helvetica,sans-serif">Wow... These questions keep popping in my head after reading this:

    Does the constant 3 hours uptime enough increase for him?

    Does the servers get flooded with large files every minute?

    Where's the mail every minute indicating the processes that are running on the server? We'll never know if there are spywares running all over the place...

    And don't forget the registry changes... Oh, and the network traffic... We can never be sure, you know...

    On second thought, perhaps these are just sent when the subjects of those emails are in critical danger, thus:
    </font><font face="tahoma,arial,helvetica,sans-serif">
    <font face="Arial">[quote user="WIldpeaks"] </font></font><font face="tahoma,arial,helvetica,sans-serif">he was going in holiday some weeks<font face="Arial">[/quote]</font>



    </font>



  • I wish it was related to his holidays but nope, that's the normal behavior of his "admin tools".



  • [quote user="xrT"]

    Where's the mail every minute indicating the processes that are running on the server? We'll never know if there are spywares running all over the place...

    <font face="tahoma,arial,helvetica,sans-serif">
    And don't forget the registry changes... Oh, and the network traffic... We can never be sure, you know...
    </font>

    [/quote]

     

    Don't give him ideas!

     

    My first thought on the uptime thing was (1) get him to put it in writing, (2) create a once-a-minute cron job (just like his!) to show uptime and how it never exceeds 3 hours, (3) show evidence to his boss and say "he's flat-out lying and should thus be terminated with extreme prejudice".  Assuming his boss believes that the uptime before the change was routinely greater than 3 hours; if not, then that's another WTF.

     

    About the only case I can think of that would justify reboots every 3 hours is if the server would otherwise overload and crash every (say) 5 hours or so, and you don't immediately have the budget to track down and fix the cause, or else buy a new server and migrate everything to it.  In this case, "uptime" would refer to percentage rather than longest-continuous-run.  But if this were the case, then surely the original poster would have said so.

     

    I'm also reminded of my wife's college music professors, at least one of whom believed that they needed to re-format their computers every month or two and re-load all the applications and documents, whether they noticed any problems or not.  And they were using Macs, mind you.

     



  • Ah there is a little confusion: he is the admin of two servers, the company's and our personal server. That's on the second that he has setup this 3-hours reboot brillantness while the evil cron jobs are on the company's. He might have setup some evil cron jobs on ours as well, but I don't know if it's the cause.

     

    > About the only case I can think of that would justify reboots every 3 hours is if the server would otherwise overload and crash every (say) 5 hours

    It kept crashing sooner and sooner after each reboot (now it doesn't even last 30 minutes, sighs). Just to check, am I the only one thinking that "fixing" a crashing server by rebooting all the time rather than actually reading the logs for finding the culprit is a wtf ?

     

    > or so, and you don't immediately have the budget to track down and fix the cause.

    Kept getting worse for monthes and it's just a simple server, no fancy programs running (afaik...), it wouldn't be hard to track down the cause.

     

    > In this case, "uptime" would refer to percentage rather than longest-continuous-run.  But if this were the case, then surely the original poster would have said so.

    Yes he meant the percentage, I was just quoting his exact words.



  • [quote user="emurphy"]

    I'm also reminded of my wife's college music professors, at least one of whom believed that they needed to re-format their computers every month or two and re-load all the applications and documents, whether they noticed any problems or not.  And they were using Macs, mind you.

    [/quote]

    For the very first time, I am ashamed to be a Mac user. ;-(



  • Now that's scary... As you're obviously talking about a Unixlike server you should expect only to reboot on patching the kernel (or libc/glibc) - I once had an uptime on a Linux box of about 1200 days!

    In terms of the space emails - what's wrong with just having boundary alerting? Panic the guy when he comes back - create a load of large files and then delete them (copy the kernel about 5000 times).



  • Well, it doesn't quite beat what one of the linux administrators at a, lets just keep it unnamed, dorm in Denmark.

    He had the great idea to set the server to notify him by email every time the logfiles changed.. Well, the idea might sound good, excetp he included the logfile which made a logentry everytime the server sent an email.

    So within a short time, the entire dorm was without internet, because he had spammed his own mail account, and brought the server to its knees in sending mails... :)



  • [quote user="Saxov"]

    Well, it doesn't quite beat what one of the linux administrators at a, lets just keep it unnamed, dorm in Denmark.

    He
    had the great idea to set the server to notify him by email every time
    the logfiles changed.. Well, the idea might sound good, excetp he
    included the logfile which made a logentry everytime the server sent an
    email.

    So within a short time, the entire dorm was without
    internet, because he had spammed his own mail account, and brought the
    server to its knees in sending mails... :)

    [/quote]

    A novel
    approach to the infinite-emails error. The more traditional method is
    two inboxes auto-forwarding to each other. +10 points for creative
    thinking.

    Also, the idea of notifying on logfile update is
    something of a WTF in itself. Isn't that like saying "tell me when you
    do anything"? Depending on the server & services in question,
    you're probably either not logging enough, or spamming yourself every
    17.3 nanoseconds.



  • [quote user="WIldpeaks"]

    You know the week's gonna be fun when your admin tells you on monday morning:
    "btw, i've set our server to reboot every 3 hours, and it seams this increases its uptime."

    [/quote]

     Are you in the Louisville, KY, USA area? My main responsibility has 400+ days right now and I happen to be on the job market :)
     



  • [quote user="RayS"]A novel approach to the infinite-emails error. The more traditional method is two inboxes auto-forwarding to each other. +10 points for creative thinking.[/quote]
    Been thinking about doing this sometime ago but sadly I have only have one mailbox with this kind of feature...




Log in to reply