Funny or interesting logging stories
-
I am writing a book on Wordpress website security and looking for funny, interesting or WTF! stories related to logging, to open that chapter. I'd love to hear any interesting tales or anecdotes highlighting the importance of logging, or how they saved your bacon at one point, especially if there is a security angle to it! I'd be happy to share a free link to the book too once its finished.
Many thanks all
-
-
Do we get commission?
-
There are rumours that this site has some sort of front page which contains funny or interesting WTF stories.
-
Isn't there a restriction on homework problems?
-
This post is deleted!
-
@Jaloopa said in Funny or interesting logging stories:
Do we get commission?
there's a generous amount of kudos on offer ;)
-
There were those times when I removed lots of logging from systems to make them go faster.
Other fun times when I worked on a massive C++ codethulian mess, where there were memory faults so adding or removing logging lines would cause the system to work differently.
-
@Gribnit said in Funny or interesting logging stories:
Isn't there a restriction on homework problems?
Been a long time since I've had to do homework. Moreso looking for interesting/cool tales from the world of security log analysis... :)
-
My grandfather used to take me logging a few times. Not on a grand scale, just a handful of trees. There was a fair amount of hacking involved too. (I was quite young still, so from what I remember I was mostly watching at that stage, and later delegated to chopping older cut-to-length logs into actual firewood with an axe.)
While those are some fond memories, I'm not sure if there are any particularly fun stories. Too young back then to remember too many details, and I'm not sure what would constitute a fun story in that context. Like, what, a tree fell on some dudes head?
-
@Carnage said in Funny or interesting logging stories:
There were those times when I removed lots of logging from systems to make them go faster.
Other fun times when I worked on a massive C++ codethulian mess, where there were memory faults so adding or removing logging lines would cause the system to work differently.One (large (production)) system (you have probably used) ran ~3X faster without the handrolled verybad logging enabled.
-
@atalltimes said in Funny or interesting logging stories:
@Jaloopa said in Funny or interesting logging stories:
Do we get commission?
there's a generous amount of kudos on offer ;)
How many exposures per kudo?
-
@Carnage said in Funny or interesting logging stories:
Other fun times when I worked on a massive C++ codethulian mess, where there were memory faults so adding or removing logging lines would cause the system to work differently.
We have such fun, except that in our case it's a mix of the size of the binary (the TEXT segment has got to fit in 32kB, which is not a lot of space) and the fact that this is soft realtime code, and logging can push things outside the timing envelope.
-
This blonde man went to Canada to seek his fortune as a lumberjack. Then he met a foreman of a logging organization who offered to give him a job.
"Now I hope you realize we expect you to cut down at least 50 trees a day," the foreman told him.
The blonde man didn't see this as a problem, so he went out and did his best. He came back sweating like a pig.
"Christ, how many trees did you cut down?" asked the foreman.
"5" He replied.
"What!? You have to do beter than that. Get up earlier tommorow." The foreman said.
So he did. Out he went with the chainsaw, he came back that night exhausted. 'How many this time?" asked the foreman.
"11" he said.
The foreman says, "That does it. I'm coming out there with you tommorow morning." The next morning, the foreman reaches the first tree and says, "This is how to cut down trees really quickly."
He pulls the rope on the chainsaw and it gives off a loud BRRRRRRRRRRUM. He notices the blonde is looking at him frantically. So he asks him what's wrong.
He replies, "What the hell is that?"
-
@dkf said in Funny or interesting logging stories:
@Carnage said in Funny or interesting logging stories:
Other fun times when I worked on a massive C++ codethulian mess, where there were memory faults so adding or removing logging lines would cause the system to work differently.
We have such fun, except that in our case it's a mix of the size of the binary (the TEXT segment has got to fit in 32kB, which is not a lot of space) and the fact that this is soft realtime code, and logging can push things outside the timing envelope.
You don't need 
HW and you don't even need messy C++ code. It's quite common (even in dead-boring Java/C# code) that adding some log messages or even just changing log level "fixes" some race condition bug.It's called Heisenbug.
-
-
Got a decent one - it's like logging but not, it has to do with serialization for inspection purposes, though.
So we were debugging a massive, massive parse, ye gods. It was geographic data. Something like, the whole Pacific ocean or something. And the run would hang, oh how it would hang. It hung... nowhere we could find, we would isolate down to the slow and it would slip out from between our fingers.
The debugger was trying to stringify the multi-GB result.
-
And @apapadimoulis thought there's no new people signing up because of mega threads.
-
@Gribnit said in Funny or interesting logging stories:
The debugger was trying to stringify the multi-GB result.
Reminds me of someone who was doing performance testing of arbitrary precision integer arithmetic, wondering why some “simple” additions were taking absolutely ages. Because it was also printing out ~10k-digit decimal numbers in his debugging code, that's why (a very expensive operation chock full of awkward divisions by 10).
-
@topspin said in Funny or interesting logging stories:
And @apapadimoulis thought there's no new people signing up because of mega threads.
We all thought that @apapadimoulis arbitrarily picked 7 topics to classify as megatopics, but clearly he knew exactly which 7 needed to be reclassified to stop people being put off.
-
@atalltimes I think you're better of fabricating whole-cloth. Security log analysis is automable to the point that intrusion-detection systems have probably been humorlessly eating all the real stories anyway.
Although, for Wordpress spifficly, there's hope. That's a backwater to the point that real tools aren't applied and there's no money to get them. The difficulty is getting someone to admit to using it, but, @boomzilla admits to using VB, so... give truth maybe a chance. maybe.
-
@Carnage said in Funny or interesting logging stories:
There were those times when I removed lots of logging from systems to make them go faster.
Other fun times when I worked on a massive C++ codethulian mess, where there were memory faults so adding or removing logging lines would cause the system to work differently.Company policy says all logging should be enabled and retained for some unreasonable amount of time. Probably written by someone who doesn't know how granular logging can get.
Correspondingly on dev, one of our products we use as a dependency (you've heard of it) needs special cleanup jobs in crontab or I'm perpetually running out of disk space.
-
@PleegWat y'know, I bet that TRACE isn't enabled for everything and that the company, once they know this, would pay you to enable it for everything. As opposed to prosecute.
-
@Gribnit said in Funny or interesting logging stories:
Got a decent one - it's like logging but not, it has to do with serialization for inspection purposes, though.
So we were debugging a massive, massive parse, ye gods. It was geographic data. Something like, the whole Pacific ocean or something. And the run would hang, oh how it would hang. It hung... nowhere we could find, we would isolate down to the slow and it would slip out from between our fingers.
The debugger was trying to stringify the multi-GB result.
I know from experience not to use vim when the (trace log) file is more than a gig or two.
-
@PleegWat said in Funny or interesting logging stories:
@Gribnit said in Funny or interesting logging stories:
Got a decent one - it's like logging but not, it has to do with serialization for inspection purposes, though.
So we were debugging a massive, massive parse, ye gods. It was geographic data. Something like, the whole Pacific ocean or something. And the run would hang, oh how it would hang. It hung... nowhere we could find, we would isolate down to the slow and it would slip out from between our fingers.
The debugger was trying to stringify the multi-GB result.
I know from experience not to use vim when the (trace log) file is more than a gig or two.
Are they using the tracing Oracle driver though? If the Oracle driver isn't tracing at its finest level, you could lose out on a lot of potential logging lines. With Oracle driver-level trace, you can stay safely around 10K log-lines per code-line executed.
-
@Gribnit Dev or prod? Dev, I don't care about what the database does as long as it works. I care about performance but that's a crapshoot on anything smaller than a proper performance testing env.
On the prod DBs I'm sure someone cares but it's not me.
-
@PleegWat said in Funny or interesting logging stories:
@Gribnit Dev or prod? Dev, I don't care about what the database does as long as it works. I care about performance but that's a crapshoot on anything smaller than a proper performance testing env.
On the prod DBs I'm sure someone cares but it's not me.
This is application level. To get the absolute most log lines out of using oracle, you need to use (and configure) the _g driver.
And it sounds like, they're explicitly demanding that you do so. So you better!
-
@PleegWat said in Funny or interesting logging stories:
@Carnage said in Funny or interesting logging stories:
There were those times when I removed lots of logging from systems to make them go faster.
Other fun times when I worked on a massive C++ codethulian mess, where there were memory faults so adding or removing logging lines would cause the system to work differently.Company policy says all logging should be enabled and retained for some unreasonable amount of time. Probably written by someone who doesn't know how granular logging can get.
Correspondingly on dev, one of our products we use as a dependency (you've heard of it) needs special cleanup jobs in crontab or I'm perpetually running out of disk space.
Our logging is
a) inconsistent
b) extremely verbose...in all the wrong places.One of our products listens on an SMTP port and injests emails, turning them into alerts for our users. A record of incoming alerts is stored on the server as a flat text file per attempt (and eventually swept to a record/storage server by a cron job) When we went to aws, it turns out that AWS's load balancers ping each endpoint periodically (like...every 30 seconds or so), sending no data. Just to check if they're alive to send traffic to.
You can guess what happened. Each of those individual tiny files grew out of control and would choke the instances after a day or so, even if there was no traffic.
Plus, I once managed to cause a temporary outage by trying to pull the logs from a production server. They were that big and that inefficiently stored that it used up all the CPU and caused the swarm manager to panic. Thankfully the failover kicked in and no traffic was lost, but it was definitely NOT A GOOD THING.
And even with all of that, the logs are basically useless. The things you'd want aren't logged, and they're choked with useless chatter.
-
@Benjamin-Hall this environment is the most secure posture available re the implementor's job.
-
@Gribnit said in Funny or interesting logging stories:
The difficulty is getting someone to admit to using it, but, @boomzilla admits to using VB, so... give truth maybe a chance. maybe.
Jam it!
Err...what?
-
@boomzilla said in Funny or interesting logging stories:
Err...what?
It's the final logging stories exam. You're naked and were just chased here by zombies.
-
@Benjamin-Hall said in Funny or interesting logging stories:
And even with all of that, the logs are basically useless. The things you'd want aren't logged, and they're choked with useless chatter.
Getting good logging set up is rather difficult. So bring a big chainsaw.
-
@Benjamin-Hall said in Funny or interesting logging stories:
And even with all of that, the logs are basically useless. The things you'd want aren't logged, and they're choked with useless chatter.
I find that very often logging is added during implementation to figure out why some things don't work quite like they should. And are left in once things do work like they should and the logging is then useless noise. Some additional logging is added in some places where it is easy to add it.
Leading to pretty useless logging overall. Any requirements to have a certain amount of logging only makes it worse.For good logging, someone needs to make a thorough analysis of possible failures, and data flows and program flows, and figure out what is important to know. And then scale down to what's really the important bits. And then add some logging for bits that turn out to have insufficient logging while it's running in prod.
-
@Benjamin-Hall said in Funny or interesting logging stories:
it turns out that AWS's load balancers ping each endpoint periodically (like...every 30 seconds or so), sending no data. Just to check if they're alive to send traffic to.
This endpoint is configurable and standard practice is that application provides special one just for this purpose (proper health check returning status according to internal monitoring, simple static file if you're lazy). Using the real production endpoint that writes something to persistent storage is
-
@Kamil-Podlesak You want the health check to go to the real service that you're monitoring, and not some proxy that can report something else unhelpful (e.g., I've seen machines where the OS kernel responded to ping but all user processes were dead because
/etc/initwas unwell). You probably need to make sure that you've got a really cheap operation that they can do without lots of authentication, but which is nonetheless served by the service. You can have infrastructure monitoring as well but it tells a subtly different story to service monitoring.
-
@dkf said in Funny or interesting logging stories:
@Kamil-Podlesak You want the health check to go to the real service that you're monitoring, and not some proxy that can report something else unhelpful (e.g., I've seen machines where the OS kernel responded to ping but all user processes were dead because
/etc/initwas unwell). You probably need to make sure that you've got a really cheap operation that they can do without lots of authentication, but which is nonetheless served by the service. You can have infrastructure monitoring as well but it tells a subtly different story to service monitoring.Yeah, the same service of course, but not the same endpoint (ie, not the same path in the url). And yes, even that by itself just means that the HTTP server is up - that's where the internal monitoring part is important.
Also, I assume that the HTTP server is part of the service process, not a proxy.
