Notifications not working
-
Not sure exactly when it started, but it seems most types of notifications have completely stopped working; I'm only getting Likes notifications, not Reply, Quote, or @Mention notifications.
Anything we can do to kickstart them @PJH and/or @boomzilla?
-
Could you be logged in from another computer? I see this a lot when I leave the notification menu open
-
Just the one. And my cyberparts, but they don't poll notifications.
Thing is, they're not in my notification history at all, as if they're not being generated:
There should be some reply notifications mixed in with those UUID likes from @accalia
-
I'm getting the same. I think this is the first post I didn't get a notification for:
https://what.thedailywtf.com/t/the-official-status-thread/1673/21232?u=boner
-
Same, I've gotten like notifications today but not replies. No other browsers/tabs open.
-
most types of notifications have completely stopped working;
Anything we can do to kickstart them @PJH and/or @boomzilla?
Just noticed this. Where did we leave the homing pigeons?
-
It's working sporadically for me, I have one like notification from about 20 minutes ago but that's it and there should be more before and after that.
Edit: Something is weird in the profiles too. I have a response that says it's from three hours ago (correct) and the latest like listed is from yesterday (wrong) and says it's 30 mins old (wrong)
-
They do seem light. This happened earlier today:
@sam...could that cause missed notifications?
-
The irony of trying to summon someone using a notification type that doesn't work
At least I got the Like notification for this post
-
Quick, someone go like Sam's posts in a pattern that encodes the topic ID in morse code!
-
I got your mention notification of me. I've gotten some reply / quote notifications, too, but not as many as I should have.
Was this maybe an attempted fix of getting dupes? And instead of squashing real dupes, you only get the first notification for a post?
-
I have a reply notification from 3 hours ago, btw.
-
Yeah, that's my last one of those, too, though I have a quote from 37 discominutes ago.
-
My last non-Like notification was around 2.5 hours ago, but I know I should have had at least a dozen if not two dozen since then
-
s around 2.5 hours
I'm quoting you. However, when I first did this, your user card opened up in the middle of the page.
-
I'm going to quote you in a post that's a reply to myself for science.
tification was around 2.5 hours ago, but I know I should have had at least a dozen if not two dozen since
-
Related (maybe): Anyone else noticed the Users column on the main topic list isn't working right either? It's not displaying the most frequent posters for certain threads; it seems to be the newer ones that are affected.
-
No notifications for either of those posts
-
Yeah...also, your "Responses" section doesn't show them.
-
Tons and tons and tons of these:
Job exception: Wrapped ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)
-
Anyone else noticed the Users column on the main topic list isn't working right either? It's not displaying the most frequent posters for certain threads;
Posting in this thread to test, since you're also the OP and Boomzilla is also the most recent poster.
-
I think sidekiq is backing up. I think I see something like 1,000 that are queued up. We may get deluged once it gets going.
-
Repro:
-
-
Uh-oh…
Just remembered: I have desktop notifications turned on…
-
-
Was this maybe an attempted fix of getting dupes? And instead of squashing real dupes, you only get the first notification for a post?
Maybe it's trying to de-dupe the Likes thread. The server must be getting red-hot if that's the case.
-
-
I'm watching the free disk space. A little bit ago, the dashboard said 15.5GB. We're now at 15.4GB. I suspect most of that is logs logging discofailure.
-
@boomzilla, it might be a good idea to post this issue as a global notice so servercooties.com picks up on it?
-
Thanks @boomzilla!
global notice notifications work!
-
OK, it's up there. For a minute I thought it wasn't going to work because it had to go through sidekiq.
-
One advantage...I don't have to take another screen shot to show you what the logs look like right now, because it's pretty much identical to this.
-
When I first looked, I think sidekiq had ~1,000 "Retries." That figure is currently approaching 1,700.
-
What's at the head of the queue? If you can see the queue, that is.
-
This is the oldest thing I can see...
Queue default Job Jobs::UserEmail.delayed_perform Arguments {:type=>:user_private_message, :user_id=>73, :notification_id=>1784307, :current_site_id=>"default"} JID 4e17aaedef02a56544585f66 Enqueued 18 days ago Retry Count 24 Last Retry about 2 hours ago Next Retry 4 days from now
This is the first thing in the Retry list:Queue default Job Jobs::PostAlert Arguments {"post_id"=>465189, "current_site_id"=>"default"} JID baeb81f1f9a06747a8424890 Enqueued 3 minutes ago Retry Count 2 Last Retry about a minute ago Next Retry less than a minute ago
Make of that what you will...
There's a lot of stuff in there.
-
A notification queue?! Why does it sound like dicksource reinvented exim/postfix?
It's fucking notifications, why does an insert to a databsae need to be queued?
-
user_id=>73
I'm intrigued about who that is…
@boomzilla said:Enqueued 18 days ago
Not going to be the cause of the notifications failing, especially as it's an e-mail action.
@boomzilla said:This is the first thing in the Retry list
That's the last post in the Bad Ideas thread (as of this posting).So I can't really make anything useful from that.
-
It's fucking notifications, why does an insert to a databsae need to be queued?
To spread the load out more evenly, obviously.
-
@boomzilla said:
user_id=>73
I'm intrigued about who that is…Looks like @galgorah
@boomzilla said:
Enqueued 18 days ago
Not going to be the cause of the notifications failing, especially as it's an e-mail action.Right. Just interesting because of the age.
@boomzilla said:
This is the first thing in the Retry list
That's the last post in the Bad Ideas thread (as of this posting).So I can't really make anything useful from that.
I have that topic set to watching. Dunno if anyone else does, but it means I'm not being told about BAD IDEAS!
-
being told about BAD IDEAS!
Clearly our quota of bad ideas have been filled, and disco is doing us a favour.
-
Ok, someone has to fix this.
I'm becoming too productive at work.
EDIT: I just finished an assignment 2 days early. Comon! They're going to expect more out of me.
-
I got a mention notification!
Just one though; the rest are still stuck
-
@RaceProUK what about this one?
-
Nope
-
And I didn't get the reply notification either.
All the counter-trolls I am missing!!!! :(
-
I think we also noticed a rogue job that was causing Sidekiq memory usage to climb high enough that it was killed by the watchdogs...
-
Yeah, I'm getting 0 [zero] as in none, niltch, notifications.
-
I'm having problems with things not being marked as read correctly.
Repro:- Enter new thread/thread with unread posts. (note position in thread)
- read until end
- close brower
- open new browser, navigate to tdwtf
- enter same thread
- reading position is now the same as in 1.
EDIT: this dows not happen 100% of the time. currently my hit rate is 50/50.
-