Notifications not working
-
@RaceProUK I'm just going to spam it with notifications to you, so that you get a full inbox when it starts working again.
-
I'd rather you didn't; I'm going to get plenty enough already
-
But you just replied to me.... so we're even so far.
Oh.... shit.
Quick, reply back.
-
I just got this reply! Are we back?
-
It's beginning to catch up, yes
-
@sam restarted sidekiq, it was "dead" (I took this to mean "processing whatever poison job is happening")
-
So Sidekiq, when it hits a job it can't complete, allows stuff to back up instead of just getting on with it? That's a bit shit.
-
Guessing that it tied up one or more of the workers, so only one or so was actually DOING THEIR JOB (note: I have no evidence for this)
-
-
oh hey, that's why testing the bot PM didn't work... it didn't get a notification until now.
-
Iiii'm gonna turn off emails for a bit
-
did my 1238 notifications break notifications? :(
-
So Sidekiq, when it hits a job it can't complete, allows stuff to back up instead of just getting on with it?
no, this is clearly a bug somewhere in the depth of sidekiq or our changes to it. the connection pool was exhausted on the process.
-
Stuff is back to normal now, you can disable the banner ... cc @PJH
-
The banner is gone.
-
Although it hung around in my session until I refreshed.
-
Same here; servercooties.com did the same (@accalia)
-
Although it hung around in my session until I refreshed.
never appeared for me until i refreshed...
Same here; servercooties.com did the same (@accalia)
that one's on @onyx unless i'm wrong.... i think that part of the page isn't refreshing?
-
goddamnit i still see the global notification about no notifications
-
Refresh
-
screw that
-
I don't think that's necessary. Just hitting your F5 should do it.
-
i tried, but F5 on my tablet didn't help.
-
Did you try restarting your tablet?
-
Oh, you're on mobile? Yeah, that's different.
-
Looks like we might have that poison task again. Notifications are being noticably slow to arrive today.
-
I'm getting notifications from May 15th
-
Yes. They are piling up again. Same problem. I think there are memory issues (i.e., using too much) on the server.
-
Yes. They are piling up again. Same problem.
lovely.......
have we summoned the relevant authorities?
-
-
have we summoned the relevant authorities?
Messages have been sent, but I think it's the wrong time of day over there.
-
Messages have been sent, but I think it's the wrong time of day over there.
what?
oh right. it's Pub-o-clock on a friday....
-
-
Sadly, I think I just missed him. When I messaged him on fail.d, he was last seen 11 minutes ago. It's now 3 hours ago. I assume he's sleeping.
-
-
I think someone (@sam) just restarted the site or something; the backlog is being cleared
-
Yeah it was me ... upgraded to latest beta as well, will check in again later today to see if sidekiq is holding up ... if it stalls again I will debug the process with rbtrace
-
I noticed the new 'long time between posts' thing is now here; I quite like it.
Next time you do an upgrade though, please please please put a global notice up first! That way we know it's coming, and it'll also appear on ServerCooties.com
-
the backlog is being cleared
Though not necessarily in order, and notifications that did appear for me are disappearing.