Getting a Node.js application which silently hangs to talk
-
Okay, currently having this problem: We're using Rocket.Chat and a problem has cropped up - the service sometimes silently hangs. As in: No errors, no warnings, simply unresponsive. Nothing in the logs and it is the application itself which has problems, the underlying server and internet connection are fine (also, everything else on the same hardware has no problems).
Since the whole things is a Node.js application, anyone have an idea where I might plug myself in to see at least why the wheels are spinning freely?
Currently doing a
tail -fon a logfile generated by a cronjob running every minute to at least get an inkling of the MTBF - because, as I said, no real errors in the logs.
-
@Rhywden tried
straceyet? I have no clue about node but this should give you a first idea whether it's still talking to anything or just stuck in awhile(1);
-
@LaoC Not yet. Will give it a whirl when it becomes unresponsive again.
-
It's been a few years now but I was using this:
https://yunong.io/2015/11/23/generating-node-js-flame-graphs/
...to try to figure out where our local server cooties were coming from.
-
@boomzilla Yeah, thanks.
In other news: This only seems to happen under semi-moderate load (or at least, whatever resource is exhausted is exhausted more quickly under load). And guess what does not happen in the afternoon and evening at school?
-
@Rhywden you'll probably find a correlation between parking lot fullness and system instability. Blocking all or most of the entrances should help.
Most likely non-facetious reason is, abandoned seats of some form are being considered in use leading to exhaustion there. Probably not anything fun like threading.
-
@Rhywden in diagnosing nodebb, we did come across a legitimate bug at one point. The other thing was a regex that went out of control somewhere occasionally. I think it still happens, but we've got enough nodes running and a watchdog process that kills the one that gets stuck. That happens a handful of times per day.
Probably not your issue, but
.
-
@Gribnit said in Getting a Node.js application which silently hangs to talk:
Probably not anything fun like threading.
It's node, so yeah...not threading, unless it's down inside of node itself (does it use threads itself?).
-
In my experience, this is usually caused by an incorrect (or completely missing) exception and event handling in a code that creates Promises. Especially working with streams is very tricky, because the adapter code needs to register all listeners and properly handle them by resolving or rejecting the Promise. If the stream fails and closes for some unexpected reason (connection broken? tcp reset?), the Promise is never resolved nor rejected - it just hangs. And there is absolutely no exception, no stacktrace, no log.
