Notifications not working


  • SockDev

    Not sure exactly when it started, but it seems most types of notifications have completely stopped working; I'm only getting Likes notifications, not Reply, Quote, or @Mention notifications.

    Anything we can do to kickstart them @PJH and/or @boomzilla?


  • I survived the hour long Uno hand

    Could you be logged in from another computer? I see this a lot when I leave the notification menu open


  • SockDev

    Just the one. And my cyberparts, but they don't poll notifications.

    Thing is, they're not in my notification history at all, as if they're not being generated:

    There should be some reply notifications mixed in with those UUID likes from @accalia



  • I'm getting the same. I think this is the first post I didn't get a notification for:

    https://what.thedailywtf.com/t/the-official-status-thread/1673/21232?u=boner



  • Same, I've gotten like notifications today but not replies. No other browsers/tabs open.



  • @RaceProUK said:

    most types of notifications have completely stopped working;

    @RaceProUK said:

    Anything we can do to kickstart them @PJH and/or @boomzilla?

    Just noticed this. Where did we leave the homing pigeons?



  • It's working sporadically for me, I have one like notification from about 20 minutes ago but that's it and there should be more before and after that.

    Edit: Something is weird in the profiles too. I have a response that says it's from three hours ago (correct) and the latest like listed is from yesterday (wrong) and says it's 30 mins old (wrong)


  • ♿

    They do seem light. This happened earlier today:

    @sam...could that cause missed notifications?


  • SockDev

    The irony of trying to summon someone using a notification type that doesn't work :rofl:

    At least I got the Like notification for this post :smile:


  • I survived the hour long Uno hand

    Quick, someone go like Sam's posts in a pattern that encodes the topic ID in morse code!


  • ♿

    I got your mention notification of me. I've gotten some reply / quote notifications, too, but not as many as I should have.

    Was this maybe an attempted fix of getting dupes? And instead of squashing real dupes, you only get the first notification for a post?


  • I survived the hour long Uno hand

    I have a reply notification from 3 hours ago, btw.


  • ♿

    Yeah, that's my last one of those, too, though I have a quote from 37 discominutes ago.


  • SockDev

    My last non-Like notification was around 2.5 hours ago, but I know I should have had at least a dozen if not two dozen since then


  • ♿

    @RaceProUK said:

    s around 2.5 hours

    I'm quoting you. However, when I first did this, your user card opened up in the middle of the page. :wtf:


  • ♿

    I'm going to quote you in a post that's a reply to myself for science.

    @RaceProUK said:

    tification was around 2.5 hours ago, but I know I should have had at least a dozen if not two dozen since


  • SockDev

    Related (maybe): Anyone else noticed the Users column on the main topic list isn't working right either? It's not displaying the most frequent posters for certain threads; it seems to be the newer ones that are affected.


  • SockDev

    No notifications for either of those posts


  • ♿

    Yeah...also, your "Responses" section doesn't show them.


  • ♿

    Tons and tons and tons of these:

    Job exception: Wrapped ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)


  • I survived the hour long Uno hand

    @RaceProUK said:

    Anyone else noticed the Users column on the main topic list isn't working right either? It's not displaying the most frequent posters for certain threads;

    Posting in this thread to test, since you're also the OP and Boomzilla is also the most recent poster.


  • ♿

    I think sidekiq is backing up. I think I see something like 1,000 that are queued up. We may get deluged once it gets going.


  • I survived the hour long Uno hand

    Repro:


  • SockDev

    @boomzilla said:

    We may get deluged once it gets going.


  • SockDev

    Uh-oh…

    Just remembered: I have desktop notifications turned on…


  • ♿

    I PM'd @sam and copied @PJH over on meta.d.


  • Fake News

    @boomzilla said:

    Was this maybe an attempted fix of getting dupes? And instead of squashing real dupes, you only get the first notification for a post?

    Maybe it's trying to de-dupe the Likes thread. The server must be getting red-hot if that's the case.


  • ♿

    :grimacing:


  • ♿

    I'm watching the free disk space. A little bit ago, the dashboard said 15.5GB. We're now at 15.4GB. I suspect most of that is logs logging discofailure.


  • SockDev

    @boomzilla, it might be a good idea to post this issue as a global notice so servercooties.com picks up on it?


  • SockDev

    Thanks @boomzilla!

    global notice notifications work!


  • ♿

    OK, it's up there. For a minute I thought it wasn't going to work because it had to go through sidekiq.


  • ♿

    @boomzilla said:

    :grimacing:

    One advantage...I don't have to take another screen shot to show you what the logs look like right now, because it's pretty much identical to this.


  • ♿

    When I first looked, I think sidekiq had ~1,000 "Retries." That figure is currently approaching 1,700.


  • SockDev

    What's at the head of the queue? If you can see the queue, that is.


  • ♿

    This is the oldest thing I can see...

    Queue default Job Jobs::UserEmail.delayed_perform Arguments {:type=>:user_private_message, :user_id=>73, :notification_id=>1784307, :current_site_id=>"default"} JID 4e17aaedef02a56544585f66 Enqueued 18 days ago Retry Count 24 Last Retry about 2 hours ago Next Retry 4 days from now
    This is the first thing in the Retry list:
    Queue default Job Jobs::PostAlert Arguments {"post_id"=>465189, "current_site_id"=>"default"} JID baeb81f1f9a06747a8424890 Enqueued 3 minutes ago Retry Count 2 Last Retry about a minute ago Next Retry less than a minute ago

    Make of that what you will...

    There's a lot of stuff in there.



  • A notification queue?! Why does it sound like dicksource reinvented exim/postfix?

    It's fucking notifications, why does an insert to a databsae need to be queued?


  • SockDev

    @boomzilla said:

    user_id=>73

    I'm intrigued about who that is…
    @boomzilla said:
    Enqueued 18 days ago

    Not going to be the cause of the notifications failing, especially as it's an e-mail action.
    @boomzilla said:
    This is the first thing in the Retry list

    That's the last post in the Bad Ideas thread (as of this posting).

    So I can't really make anything useful from that.


  • SockDev

    @delfinom said:

    It's fucking notifications, why does an insert to a databsae need to be queued?

    To spread the load out more evenly, obviously.


  • ♿

    @RaceProUK said:

    @boomzilla said:
    user_id=>73

    I'm intrigued about who that is…

    Looks like @galgorah

    @RaceProUK said:

    @boomzilla said:
    Enqueued 18 days ago

    Not going to be the cause of the notifications failing, especially as it's an e-mail action.

    Right. Just interesting because of the age.

    @RaceProUK said:

    @boomzilla said:
    This is the first thing in the Retry list

    That's the last post in the Bad Ideas thread (as of this posting).

    So I can't really make anything useful from that.

    I have that topic set to watching. Dunno if anyone else does, but it means I'm not being told about BAD IDEAS!



  • @boomzilla said:

    being told about BAD IDEAS!

    Clearly our quota of bad ideas have been filled, and disco is doing us a favour.



  • Ok, someone has to fix this.

    I'm becoming too productive at work.

    EDIT: I just finished an assignment 2 days early. Comon! They're going to expect more out of me.


  • SockDev

    I got a mention notification! :smile:

    Just one though; the rest are still stuck :rolleyes:



  • @RaceProUK what about this one?


  • SockDev

    Nope



  • And I didn't get the reply notification either.

    All the counter-trolls I am missing!!!! :(



  • I think we also noticed a rogue job that was causing Sidekiq memory usage to climb high enough that it was killed by the watchdogs...



  • Yeah, I'm getting 0 [zero] as in none, niltch, notifications.



  • I'm having problems with things not being marked as read correctly.
    Repro:

    1. Enter new thread/thread with unread posts. (note position in thread)
    2. read until end
    3. close brower
    4. open new browser, navigate to tdwtf
    5. enter same thread
    6. reading position is now the same as in 1.
      EDIT: this dows not happen 100% of the time. currently my hit rate is 50/50.


  • :fearful:


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.