Wooden table approach to logging



  • There are many ways to do logging. In Java, one common way is with Apache log4j. It's trivial to use, minimally invasive and reasonably low overhead. Just set up a config file to specify what log levels you want to capture, in what file to put them and go.

    In our application, we have 13 (thirteen) log file appenders. They all dump into the same physical file, but with different output formats. Then we have 26 different log file parsers (two for each appender format for products A and B), each of which is run independently to scan the logs, capture the relevant log lines, parse out 20-ish fields (depending upon the appender/product), and write them to assorted database tables. Now some of our log files approach 10GB (and span multiple days), which means 260GB worth of parsing, just to capture the subset representing the current day logs.

    I queried: why not just put the data in some object at log-time and queue that up for a low priority background task (to dump to the db) in the application. This way, if someone changes the spelling on some log statement, the whole brittle mess doesn't come crashing down, and oh BTW, we don't need to parse 260GB of crap just to get 10K log statements.

    The answer: Threading is hard to do, and there is a lot of overhead in putting a message on a queue.

    WTF?! - off the top of my head (haven't tried to compile, but yes, it is just this simple):

    class MyDbQ<t><T> extends Thread {
    private final SynchronousQueue<t> queue;<t>
    </t></t></t> private final static MyDbQ instance = new MyDbQ();

    <t><t><t> private MyDbQ() {
    queue =</t></t></t> new SynchronousQueue(true); // true=fifo ("fair")
    <t><t><t>    start();
    }

    public static MyDbQ getInstance() {
    return instance;
    }

    // MyDbQ.getInstance().add(...)
    public void add(final T item) {
    queue.offer(item);
    }

    public void run() {
    while (true) {
    try {
    final T item = queue.take();
    // process item here</t></t></t>
    <t><t><t>      } catch (InterruptedException e) {</t></t></t>
    <t><t><t>        // handle error</t></t></t>
    <t><t><t>      } catch (Exception e) {
    // handle error
    }
    }
    }
    }
    </t></t></t>

     



  •  log4j supports rotating file appenders out of the box, and that seems like a simpler solution.

     (PS it's bad style to extend Thread, and even worse style to call Thread.start() in a constructor. You should implement Runnable and  start your thread elsewhere).



  • @pjt33 said:

     log4j supports rotating file appenders out of the box, and that seems like a simpler solution.

     (PS it's bad style to extend Thread, and even worse style to call Thread.start() in a constructor. You should implement Runnable and  start your thread elsewhere).

     

    Elsewhere as in ...

    WHAT IF - Used in simulation languages. Branches before evaluating test conditions.

    OR ELSE - Conditional threat, as in: "Add these two numbers OR ELSE!"

               </p><p>WHY NOT? Executes the code that follows in a devil-may-care
                 fashion.
                 
               </p><p>WHO ELSE? - Used for polling during I/O operations.
                 
               </p><p>ELSEWHERE - This is where your program really is when you think
                 it's here.
                 
            </p>GOING GOING GONE - For writing unstructured programs. Takes a
                 random branch to another part of your program. Does the work of 10
                 GOTOs.<p>(Taken from <a href="http://www.tlc-systems.com/babbage.htm">Babbage)</a></p><p>&nbsp;</p>


  • Actually, log4j provides a JDBCAppender and you know what this means. I haven't used, ever, but for some cases it might be a good idea. Also, you can use the JMSAppender or the AsyncAppender to do exactly what you're proposing. I really like this http://code.google.com/p/log4j-snmp-trap-appender/



  • All: I agree, there are a whole bunch of ways to do this, all of which are better than what's there. I threw that example together in about 2 minutes BC (before coffee) just to illustrate how trivial it is; using the log4j stuff is even more trivial in terms of code, but involves configurations (see my post from a day or so ago: 1920 configuration files, and counting).



  • @renewest said:

    (Taken from Babbage)
     

    As our namesake, we chose Charles Babbage, who died in poverty while trying to finish building the first computer. The new language is thus named after the first systems designer to go over budget and behind schedule.

    Ha!



  • As our namesake, we chose Charles Babbage, who died in poverty while trying to finish building the first computer. The new language is thus named after the first systems designer to go over budget and behind schedule.

    Not to mention  suffering major feature creep.



  • On a tangentially related note, at my last job we made a product with a browser-based UI. We made a crash handler with a (fixed-size, scrolling) textarea containing the backtrace, so that when there was a crash our customers could copy the text out of there and email it to us. (We couldn't get to the logs ourselves; the software ran on customer networks.)

    ... naturally, we ended up just getting lots of screenshots of crash pages with only a portion of the backtrace visible. Usually .bmp files, to boot. We had to modify the page so that it wasn't in a textarea anymore, and adjust the font sizes so as much of the backtrace fit on the screen as possible...



  • @fennec said:


    ... naturally, we ended up just getting lots of screenshots of crash pages with only a portion of the backtrace visible. Usually .bmp files, to boot. We had to modify the page so that it wasn't in a textarea anymore, and adjust the font sizes so as much of the backtrace fit on the screen as possible...

    Just encode the data in RGB values, and you can fit several megabytes of it on the screen. Problem solved! Better make sure your customers never discover JPEG though...



  • Deja vu! I had the same conversation with a collegue at my last employer. He argued that they employed log scraping since there "is no other way to find out if an async process is finished". They used this approach when the customer complained enough about performance that they had to throw out their "worst case sleep" method of waiting for completion. And these guys were postgraduates. The horrors!


Log in to reply