Constant 503 errors make Dicsourse unusable, break infinite scrolling worse than it was already broken.


  • :belt_onion:

    I can understand rate-limiting things to stop the bots. But I am using the forum as a regular user and ALL I FUCKING GET ARE CONTINUOUS FUCKING 503 ERRORS. Apparently the 503's are not actually from rate-limiting, just a frequently occurring side-effect that started after the rate-limiting was put in place.

    Also, when it 503's the part where Dicsourse tries to load the next batch of posts in infiniscroll, the entire fucking thing is just broken. It won't load the next batch and it won't retry. You have to scroll up an arbitrary amount and then scroll back down. This basically happens to me on EVERY MOTHER FUCKING SCROLL ATTEMPT NOW.

    MAYBE MAKE YOUR FORUM WORK PLEASE?

    Thanks.

    (image from infiniscroll shitting itself trying to load next post batch)



  • @darkmatter said:

    I can understand rate-limiting things to stop the bots. But I am using the forum as a regular user and ALL I FUCKING GET ARE CONTINUOUS FUCKING 503 ERRORS.

    Rate limiting is 429 or so. Though yeah, when something goes wrong with a request, you're pretty much in la-la-land as far as Discourse is concerned. You might get a "something went wrong" dialog, or you might just screw things up in millions of ways.


  • :belt_onion:

    503 is what I get for liking too much too fast.


  • :belt_onion:

    Also, wtf is this shit? How is that 2 posts?



  • @darkmatter said:

    503 is what I get for liking too much too fast.

    Then it's not actual rate limiting, it's more of a "shit goes down" rate limiting.


  • :belt_onion:

    @Maciejasjmj said:

    it's more of a "shit goes down" rate limiting

    ah, so basically dicsourse has been completely unusable for the last week due to "bot rate limiting fixes" that rate limited the bots and also broke the rest of the site to go with it. gotcha.



  • Well if Jeff didn't go boast about how bot proof the forums were in the first place, he wouldn't be learning his lesson now.


  • FoxDev

    copypasta from a different thread that i thought was this one when i replied:

    @accalia said:

    as for the 503s.... no idea whats causing that. it sure as heck ain't my bots as when the site goes down they terminate (upon receiving even one 5xx response they will terminate.and wait to be manually restarted)



  • This post is deleted!

  • Banned

    Might be related to the very ill advised change in page size from 20 to 50.



  • @codinghorror said:

    Might be related to the very ill advised change in page size from 20 to 50.

    That was done weeks ago - long before this.


  • Banned

    Yes but the nginx rate limiting in the template is relatively new. (We recently made these rate limits default as well).



  • @codinghorror said:

    Yes but the rate limiting in the template is relatively new.

    So... this is a regression?

    Wait, though... if the page size is smaller, doesn't that mean there will be more requests, not fewer because you'll be loading slices 5 times per 100 posts than 2 times per 100 posts?



  • Get out of here with your logic. I bet it loads the 20/50 topics in sets of 5.



  • @delfinom said:

    Get out of here with your logic. I bet it loads the 20/50 topics in sets of 5.

    Actually, I don't believe it does... but yes, my logic is clearly inappropriate here.


  • FoxDev

    well don't forget the pages overlap. I;'m nto sure why they overlap, you'd have to ask Jeff, but they do.

    so with 50 loading 100 posts would probably generate 5 requests and with a page size of 20 would likely generate 11.



  • @accalia said:

    well don't forget the pages overlap. I;'m nto sure why they overlap, you'd have to ask Jeff, but they do.

    so with 50 loading 100 posts would probably generate 5 requests and with a page size of 20 would likely generate 11.

    Either way: smaller page size seems to logically correlate with more requests - which for a rate limiter is a serious problem in my mind.


  • FoxDev

    not disagreeing with your logic. in fact agree with it 100%, disagree with your numbers.

    ;-P



  • @accalia said:

    not disagreeing with your logic. in fact agree with it 100%, disagree with your numbers.

    ;-P

    The numbers you have are probably no more right than the numbers I have because, you know...


  • FoxDev

    This post is deleted!


  • @accalia said:

    discoursistency?

    I was trying to be nice to the CDCK Overlords while they were possibly listening to us...


  • FoxDev

    ah, right. we're supposed to being nice....

    In the spirit of being nice: comment withdrawn.



  • @accalia said:

    ah, right. we're supposed to being nice....

    In the spirit of being nice: comment withdrawn.

    We're trying but sadly I feel it's not working. Takes two to tango.


  • FoxDev

    Well, two can't tango if one gives up. we should continue the effort to be nice in the hopes that eventually the [anal sphincters] on both sides will eventually stop being so mean to each other.

    so we two shall be nice and encourage all the others to do the same!



  • @accalia said:

    Well, two can't tango if one gives up. we should continue the effort to be nice in the hopes that eventually the [anal sphincters] on both sides will eventually stop being so mean to each other.

    so we two shall be nice and encourage all the others to do the same!

    I'll try but I suspect I won't try too hard any further 😦


  • Discourse touched me in a no-no place

    @codinghorror said:

    Yes but the nginx rate limiting in the template is relatively new. (We recently made these rate limits default as well).

    So it's related to this, not the increase in 'page' size, then.



  • @delfinom said:

    if Jeff didn't go boast about how bot proof the forums were in the first place, he wouldn't be learning his lesson now.

    There's nothing to learn. The bots are doing it wrong.



  • Infiniscroll has just gone totally broken for me. I was having some ERR_CACHE_MISS errors just a while ago, and now nothing - the request isn't even made, I just scroll to the bottom and nothing happens.

    Tested on Chrome and Firefox.

    http://i.imgur.com/8Y3XTXE.png


  • BINNED

    @Maciejasjmj said:

    Infiniscroll has just gone totally broken for me. I was having some ERR_CACHE_MISS errors just a while ago, and now nothing - the request isn't even made, I just scroll to the bottom and nothing happens.

    Have this too at several topics


  • kills Dumbledore

    @codinghorror said:

    Might be related to the very ill advised change in page size from 20 to 50.

    if it's so ill advised, why is it even an option?



  • Because you are Doing it Wrong (tm).



  • God dammit arantor, quit talking like Jeff. If a bug was just introduced, it's a bug, not a regression. A regression implies an issue came back.



  • @Matches said:

    God dammit arantor, quit talking like Jeff. If a bug was just introduced, it's a bug, not a regression. A regression implies an issue came back.

    I was trying to talk to Jeff in Jeffspeak - because I figure if there's any hope of getting him to listen I have to talk to him on his terms, not ours.



  • @Arantor said:

    That was done weeks ago - long before this.

    Wasn't that months ago?



  • @Arantor said:

    That was done weeksmonths ago - long before this.

    FTFY



  • @darkmatter said:

    I can understand rate-limiting things to stop the bots. But I am using the forum as a regular user and ALL I FUCKING GET ARE CONTINUOUS FUCKING 503 ERRORS. Apparently the 503's are not actually from rate-limiting, just a frequently occurring side-effect that started after the rate-limiting was put in place.
    <img src="/uploads/default/8287/95327b09cd61b300.png" width="690" height="66">

    Also, when it 503's the part where Dicsourse tries to load the next batch of posts in infiniscroll, the entire fucking thing is just broken. It won't load the next batch and it won't retry. You have to scroll up an arbitrary amount and then scroll back down. This basically happens to me on EVERY MOTHER FUCKING SCROLL ATTEMPT NOW.

    MAYBE MAKE YOUR FORUM WORK PLEASE?

    Thanks.
    <img src="/uploads/default/8289/a33f3182e787ac9c.png" width="690" height="16">
    (image from infiniscroll shitting itself trying to load next post batch)

    I should start monitoring my network traffic when I post. Discourse keeps freezing on me right when I post and I wonder if it's related.

    Edit: Of course it doesn't happen when I'm watching.


  • :belt_onion:

    @codlnghorror said:

    Might be related to the very ill advised change in page size from 20 to 50.

    That makes so little sense. Infiniscroll fails to scroll randomly when new posts are attempted to be fetched and fail due to some kind of botched rate-limiting server patch... so we should go back down to 20 posts between loads, causing them to load them MORE frequently as the solution? Are you trying to reverse psychology the server into working better?



  • Not only that, but apparently he knows better on the matter than everyone. Because it's not only ill advised. It's very ill advised.



  • To say nothing of the fact that 50 posts is not very much. The standard for any other (PHP, hell-stew, whatever else) forum is 50 posts to a page.



  • In fact, I think that's why our page size was bumped to 50: we were getting sick of the puny 20 post page size, so we got our admins to bump it to a more reasonable 50.



  • It's probably N requests per X time

    If you drop it to 20 rather than 50, it's 20discourse requests / time it takes you to scroll down to force 20discourse requests again.

    If you're checking on a cycle time of say, 200ms, it can explain this thought process.

    No. I'm not excusing the lunacy of it.



  • @Arantor said:

    my logic is clearly inappropriate here.

    FTFY


  • Banned

    Just checking the default rate limiting template in the image, it is 10 requests in a second and 100 per minute from the same IP. The 100 per minute strikes me as a little low for a default.


  • Discourse touched me in a no-no place

    @codinghorror said:

    The 100 per minute strikes me as a little low for a default.

    A little, yes.


  • FoxDev

    thanks for the pointer there.

    Maths Tiem!

    @sockbot enforces a 250ms delay after any request to discourse using a busy wait and as she is single threaded that means that a single instance of sockbot is rate imited to no more than 4 requests/second.

    therefore at most 3 instances can share an IP and be guaranteed to never exceed the rate limits.

    but wait! there's more!

    @sockbot, unlike the web browser does not constantly poll /message-bus/uuid4/poll. there is a configurable delay (governed by the cyborg switch) of 20 or 60 seconds between the completion of processing one poll of the message bus before requesting another poll. This means that sockbot is now down to an average of <<1 post a second.

    @sockbot also polls notifications.js on a cycle of 30/90 seconds (again depending on cyborg mode (cyborg mode multiplies all delays by 3)

    hmm... i should be able to run 10 or so bots comfortably without hitting the rate limits.


    and i'll agree with @loopback0 and @codinghorror that 100/minute is a bit restrictive. it shoudl probably be somewhere around 720 if we are going to allow sustained 12 requests per second, or somewhere around 500 if we're going to throttle sustained down a bit.



  • @accalia has summoned me, and so I appear.



  • @sockbot
    Dance!


  • Discourse touched me in a no-no place

    @codinghorror said:

    Just checking the default rate limiting template in the image, it is 10 requests in a second and 100 per minute from the same IP. The 100 per minute strikes me as a little low for a default.

    Given the polling I'm assuming the pages are doing naturally, what's the rough rate for (say) 10 tabs open on different topics?


  • :belt_onion:

    If that's true, then what's breaking dicsourse. I get constant 503s just reading one page and doing nothing the fuck else at all, and surely am nowhere near touching 100 requests per minute just reading a single fucking page... turns on network logging oh wait... how the fuck... I AM using over 100 requests per minute just to scroll through reading a single fucking thread? Avatars, smilies, images don't even count against your limit because I tested that explicitly. So how the fuck is scrolling the page triggering 100 requests a fucking minute? Well, when you're sitting idle, it only polls once every 12 seconds. When you scroll, for some reason, it polls basically between every single timings post. Which it seems to post a set of timings every second if I scroll, so it then polls once per second too. Hmm let's add that up... oh 120requests/minute. And bye bye dicsourse.

    Maybe you just need someone that doesn't read like a turtle to be at the helm of the rate-limiting, because surely half the people here are struggling to read ANY fucking topics anymore with this asinine limit. While I can read fairly fast, I'm not a speed reader, and I always check the blue dots to make sure they are vanishing. Which one would think implies that I must be Doing It Right™ right? I mean, that's Dicsourse's approved this is how long it takes to "read" a post time, right?? So how is it that by reading posts at the RATE SPECIFIED ACCEPTABLE BY THE SYSTEM, I am flooded with 503 Service Unavailable brokenness after I get through about the first minute of reading a thread? And then every other post's blue dot stops disappearing. Then the next set of posts fails to load entiretly and Dicsourse is completely unable to recover from it without an F5.

    Until that shit's fixed, there's no point to my reading any Dicsource forum. They're completely and totally fucking broken. Also, congratulations on 503'ing your own error reporting services:

    Totals: 181 requests in somewhere between 90 - 120 seconds, all dicsourse browser generated, all by a human reading every single word of content.


  • :belt_onion:

    @PJH said:

    Given the polling I'm assuming the pages are doing naturally, what's the rough rate for (say) 10 tabs open on different topics?

    Just tested it. One poll every 12 seconds per tab - so opening topics in new tabs = unusable forums at about 8 - 9 tabs even if you can only read at a rate of 1 word per minute.

    Hell, any nominally fast reader can spam themselves off dicsourse as I am having happen to me just by reading a single topic in a single tab and nothing else. See math in prior post.


Log in to reply