We need to monitor the heap size



  • After our peer team rehacked their system to save less data, they started complaining that their system had experienced a significant performance issue. They were running JProbe to try and analyze the performance problem, but just couldn't figure it out. Since I'm the resident threading expert (at least around here), they asked for my help. I found this being called after every single logical transaction:

       public void saveHeap() {
    // grab max heap limit, current heap size and free bytes from Runtime
    // store info to a logging table in the db, for subsequent analysis
    }

    When I pointed it out, they insisted that it wasn't the problem as it was just being run once a transaction, and in straight-line code.

    I pointed out that this straight-line code was being run in a thread, which was one of 100 threads, all doing the same thing in parallel.

    I looked in the log table and there were tens of millions of rows.

    ...But we need to monitor heap size!

     



  • Why is there a table in the DB that allows them to do this? That's just asking for trouble.



  • @Evilweasil said:

    Why is there a table in the DB that allows them to do this? That's just asking for trouble.

    They wanted to be able to store it for longer than the log rotation period, so just logging the data wasn't sufficient.

    I don't mind them storing it in the db. I don't even mind them taking a snapshot after every transaction, but every thread is reporting the same information - they could cut out 99.999% of the overhead by just doing it once a minute on a timer in its own thread. These guys just don't comprehend parallel processing.

     



  • If they fail to comprehend parallel processing, perhaps they should only be entrusted with one thread.

    Or maybe a mop. You can tell them I said that, too.



  • @Evilweasil said:

    If they fail to comprehend parallel processing, perhaps they should only be entrusted with one thread.

    Or maybe a mop. You can tell them I said that, too.

    Nah, a mop has too many threads. they'd get lost trying to put it in the bucket.


  • @Steeldragon said:

    Nah, a mop has too many threads.

    +1



  • Snoofle, I've got a question for you.  How do you avoid workplace violence?



  • @locallunatic said:

    Snoofle, I've got a question for you.  How do you avoid workplace violence?

    How do you know he does? Maybe he's just got really good at hiding the evidence of said violence...

     



  • @locallunatic said:

    Snoofle, I've got a question for you.  How do you avoid workplace violence?

    I stopped taking other peoples' stupidity personally a long time ago. For me, it's the satisfaction of fixing the mess; openly mocking the idiot is a bonus.


  • What I don't understand is why Alex Papadimoulis doesn't open a Snoffle category and lets you post your articles there on the front page.



  • @snoofle said:

    These guys just don't comprehend parallel processing.
     

    It's kind of like recursion. To understand recursion you have to, uh, well, there's that calling stuff and, you know, some complex stuff, and uh, you can't do that in this language so it doesn't matter so just forget about it.

     

     



  • @TheRider said:

    What I don't understand is why Alex Papadimoulis doesn't open a Snoffle category and lets you post your articles there on the front page.

    Because the front page is like Oracle. Whenever something new is assimilated by the moloch, the thing withers, loses colour, and becomes incoherent zombie-like shadow of its past self.



  • @SteamBoat said:

    It's kind of like recursion. To understand recursion you have to, uh, well, there's that calling stuff and, you know
     

    You can learn more about recursion here.


  • BINNED

    @bannedfromcoding said:

    @TheRider said:
    What I don't understand is why Alex Papadimoulis doesn't open a Snoffle category and lets you post your articles there on the front page.
    Because the front page is like Oracle. Whenever something new is assimilated by the moloch, the thing withers, loses colour, and becomes incoherent zombie-like shadow of its past self.
    Are you saying we should bring back "Mandatory Fun Day"?

    I'm starting a petition then.



  •  Snoofle could be the next "House MD", just replace the medical nonsense words with software nonsense words.

    "It's not Lupus" -> "It's not a Rootkit"

     



  • Oh my God!

    What are the lines of thinking to reach this conclusion? Didn't ANY developer from that team thought that it was a VERY VERY VERY stupid idea? Do they know nothing about async queues, logging rotation, threading, database performance?

    I've seen WTFs here which are plain stupid, but this has reached a point of epic fail, because, it doesn't even work! Well, the heap is shared by all thread, so having 100's of thread dumping its size is totally irrelevant.

    As an added bonus, @snoofle, did someone ever looked at those metrics? Bet no.

    For future reference on a way of doing this EASILY, there's this thing called JMX, read about it, learn it and use it.



  • @snoofle said:

    These guys just don't comprehend parallel processing.

    Parallel processing is a form of schizophrenia.  Imagine having 100 little gremlins living inside your brain. Could become quite exciting!

     - I'm schizophrenic, and so am I. Me too!

     



  • @bgodot said:

     Snoofle could be the next "House MD", just replace the medical nonsense words with software nonsense words.

    "It's not Lupus" -> "It's not a Rootkit"

     


    Maybe it's a segfault? "It's never a segfault!"



  • @Cassidy said:

    @SteamBoat said:

    It's kind of like recursion. To understand recursion you have to, uh, well, there's that calling stuff and, you know
     

    You can learn more about recursion here.

    That's not really it.  You can actually learn more about recursion here.

     



  • @DaveK said:

    @Cassidy said:
    @SteamBoat said:
    It's kind of like recursion. To understand recursion you have to, uh, well, there's that calling stuff and, you know
    You can learn more about recursion here.
    That's not really it. You can actually learn more about recursion here.

    Here's a puzzler: which joke is least funny? This "link to the same thread and call it a manual on recursion" joke? (Bonus points for making the EXACT SAME "JOKE" twice in the same thread.) Or the "replace words in Shakespeare with computer terms" joke?

    The answer is if you found either funny, seppuku immediately.

    Protip: seppuku is inherently funny.



  • @blakeyrat said:

    @DaveK said:
    @Cassidy said:
    @SteamBoat said:
    It's kind of like recursion. To understand recursion you have to, uh, well, there's that calling stuff and, you know
    You can learn more about recursion here.
    That's not really it. You can actually learn more about recursion here.

    Here's a puzzler: which joke is least funny? This "link to the same thread and call it a manual on recursion" joke? (Bonus points for making the EXACT SAME "JOKE" twice in the same thread.) Or the "replace words in Shakespeare with computer terms" joke?

    The answer is if you found either funny, seppuku immediately.

    Protip: seppuku is inherently funny.

    I didn't find either funny, but in general I'm smart enough to not read, or just skim, threads that I don't enjoy reading.

  • Discourse touched me in a no-no place

    @Sutherlands said:

    I didn't find either funny, but in general I'm smart enough to not read, or just skim, threads that I don't enjoy reading.
    You mean we're not supposed to continually gripe, moan and whine at everything we don't find funny on here? Oh dear, someone's doing it wrong then...


  • BINNED

    @Sutherlands said:

    @blakeyrat said:
    Here's a puzzler: which joke is least funny? This "link to the same thread and call it a manual on recursion" joke? (Bonus points for making the EXACT SAME "JOKE" twice in the same thread.) Or the "replace words in Shakespeare with computer terms" joke?

    The answer is if you found either funny, seppuku immediately.

    Protip: seppuku is inherently funny.

    I didn't find either funny, but in general I'm smart enough to not read, or just skim, threads that I don't enjoy reading.
    You're probably also smart enough to learn more about operating systems and programs you're expected to work with to find information that will make your life easier.


  • @toon said:

    @bgodot said:

     Snoofle could be the next "House MD", just replace the medical nonsense words with software nonsense words.

    "It's not Lupus" -> "It's not a Rootkit"

     


    Maybe it's a segfault? "It's never a segfault!"

    "Except when it's a segfault".


  • Considered Harmful

    @blakeyrat said:

    Here's a puzzler: which joke is least funny? This "link to the same thread and call it a manual on recursion" joke? (Bonus points for making the EXACT SAME "JOKE" twice in the same thread.) Or the "replace words in Shakespeare with computer terms" joke?

    The answer is if you found either funny, seppuku immediately.

    Protip: seppuku is inherently funny.

    The recursion joke might have been funny if Google hadn't beat them to the punch.

    Some of the Shakespeare quotes, on the other hand, had me audibly laughing in my cubicle.



  • Sorry, I tried to resist, but it just kept eating at me:

    @snoofle said:

    Since I'm the resident threading expert (at least around here)

    That is what "resident" means, yes ;-)

     



  •  In the same vein... 

    I had written a bit of code to display the status of various servers and plant machinery on a web page. It was mostly a simple affair, ping some machines, check if the FTP server was running, that the ISDN line was up, etc. The last bit, the plant machinery, was a pain. Some of them were 1970's relics, and the only info they gave out was a series of lights on their panel. I solved that with a bunch of PCs from the scrap heap and a mix of DACs made from Radio Shack parts, DPST relays, phototransistors, and black and white Logitech cameras pointed at the panels. It worked, but the fastest you could update some of the info with known fresh stuff was every 15-20 seconds. Once a minute was fine, I was told. After leaving them a pile of spare parts and documentation, I took my check and departed.

    A year or so later, I get a call. Some of the plant machinery wasn't reporting in. I take a look remotely, and someone had mucked with the config file so that it tried to update every five seconds. Reverted the change, explained what happened, and sent a bill.

    Two weeks go by, and in comes a second call. Nothing was working, and it had taken down their fileserver too! This time I drove over to have a look and smack the hand of whomever broke it. 

    Someone's nephew had been in to redesign the company webpage, and thought it would be cool to snazzify the status page and split it onto different versions. One for management, showing everything with uptime graphs, one for plant employees, showing a red/yellow/green board readable across the room, one for customers, complete with a little diagram of the plant and animated arrows showing the expected movement of things on the plant floor, and one each for dumbphones and Palm devices. It looked pretty damned spiff for the early naughties.

    The goof?  After breaking my status service once by making it poll too fast, and then discovering he didn't understand how I was generating the HTML with a mix of C and shell script, he'd gone with plan B; scrape the old web page and turn it into something his new code could read easily. But he was doing it five times, once for each version, keeping super-insane logs (Opening page for scrape at $time! Scrape successful! Retrieved $bytes in $seconds at $time! Mangling the scrape at $time!, etc), and doing it as fast as he possibly could. He was swamping the poor 486 that ran the status service and had filled the file-server with garbage logs.

    At least his java was readable when it came time to fix it.



  • @snoofle said:

    Since I'm the resident threading expert (at least around here)

    I suspect that to achieve a relative level of "Expert" at your place of work, you would need to know what a thread is. Or, maybe just how to spell "Thread". If you actually know how to use them, that elevates you to somewhere north of Godhood.



  • @blakeyrat said:

    @DaveK said:
    @Cassidy said:
    @SteamBoat said:
    It's kind of like recursion. To understand recursion you have to, uh, well, there's that calling stuff and, you know
    You can learn more about recursion here.
    That's not really it. You can actually learn more about recursion here.

    Here's a puzzler: which joke is least funny? This "link to the same thread and call it a manual on recursion" joke? (Bonus points for making the EXACT SAME "JOKE" twice in the same thread.)

    You're not very observant, but I guess that's only to be expected when you go for sheer post quantity over quality.  Cassidy's post links to an earlier post in the thread, and is thus not recursive.  My post links to itself, and therefore illustrates recursion better.



  • @DaveK said:

    @blakeyrat said:

    @DaveK said:
    @Cassidy said:
    @SteamBoat said:
    It's kind of like recursion. To understand recursion you have to, uh, well, there's that calling stuff and, you know
    You can learn more about recursion here.
    That's not really it. You can actually learn more about recursion here.

    Here's a puzzler: which joke is least funny? This "link to the same thread and call it a manual on recursion" joke? (Bonus points for making the EXACT SAME "JOKE" twice in the same thread.)

    You're not very observant, but I guess that's only to be expected when you go for sheer post quantity over quality.  Cassidy's post links to an earlier post in the thread, and is thus not recursive.  My post links to itself, and therefore illustrates recursion better.

    How's this for observant: we get the joke, it's extremely lame, and no matter which post you refer to, we'll still get it, and it will still be at least as lame as if you'd referred to any other post in existence.



  • @DaveK said:

    Cassidy's post links to an earlier post in the thread, and is thus not recursive. My post links to itself, and therefore illustrates recursion better.

    And both are predictable and extremely unfunny. (Actually, as far as jokes go, a predictable punchline is basically the definition of "unfunny".) That's what I was commenting on. I'm sure the pedantic dickweeds appreciate that your "joke" meets the pedantic dickweed acceptance criteria; I only care that it's a fucking awful joke.



  • @DaveK said:

    My post links to itself, and therefore illustrates recursion better.
    Your post links to itself, and therefore illustrates an infinite loop. Good day sir.

     



  • @Zylon said:

    @DaveK said:

    My post links to itself, and therefore illustrates recursion better.
    Your post links to itself, and therefore illustrates an infinite loop. Good day sir.

     

    Nope, that's infinite recursion. Please read and follow the instructions in this sentence.



  • @Ben L. said:

    Please read and follow the instructions in this sentence.
    Ah, self-documenting code!

     



  • @blakeyrat said:

    @DaveK said:
    Cassidy's post links to an earlier post in the thread, and is thus not recursive. My post links to itself, and therefore illustrates recursion better.

    And both are predictable and extremely unfunny. (Actually, as far as jokes go, a predictable punchline is basically the definition of "unfunny".) That's what I was commenting on.

    You're not very observant, but I'm surprised that you don't even pay attention to your own posts. (Actually, I suppose that's also only to be expected when you go for sheer post quantity over quality.)

    @blakeyrat said:

    (Bonus points for making the EXACT SAME "JOKE" twice in the same thread.)

    That's what I was commenting on.  I wasn't "making the exact same joke", I was correcting a joke that, regardless how unfunny you find it, had been mis-told.

    Now, tell me again how unfunny you find it, and watch me care. 


Log in to reply