Constant 503 errors make Dicsourse unusable, break infinite scrolling worse than it was already broken.
-
empiraclly i'm getting about a request a second per tab, +/- 20% being statically open.
if the /timings posts continure while not at top of z order that will be about 10 request/second for ten tabs or ~600 requests a minute. you should be able to keep that up for 10 seconds before getting rate limited.
not entirely sure how the rate limiting is enforced. if it's just a delay before processing (so you stay limited but no errors) then you'll just notice sluggish response from discourse. if it replies with 429 or other status you'll start seeing tabs break.
if /timings stop wnen not active tab then the load should be much lighter.
but then i just use this software i'd defer to a disco dancing dev if they had a better answer.
-
I just opened up the last 10 topics I wanted to browse, as I would operate on any other site I use. Check out the traffic and unusability below.
Congrats, if the goal was to find foundations of web browsing mechanics to break, Dicsourse has done it again. First native scrollbars and links were trashed, and now tabbed browsing was done away with altogether. I look forward to the new and innovative ways in which pre-existing browser features can be crippled.
-
And this is why I have one or maybe - MAYBE - two tabs open.
-
I always have two tabs open — one for whatever topic I'm actively reading, and one for the emoji FAQ. I don't generally get 429 or 503 or whatever, except in that topic, or when Discourse is just generally soiling its diaper. Sometimes I open a third, to my profile and/or notification page, so I can see notifications older than the 10 most recent.
-
Any plugins or anything like that? I confirmed these rate limits do not apply to requests for static content.
Edit: we set it to 200 requests for dynamic content per minute here. Just to confirm, so the main way to repro this is to have a ton of tabs (8+) open? No special behavior other than "tons of tabs?"
-
a ton of tabs (8+) open
100 or 200 is 'a ton.' 8 certainly isn't, and I regularly open 5-10 at a time to catch up on unreads.
Unless it's my desktop with 1.5G of memory, which craps itself with more than 4 (DC) tabs long before 503's could be an issue. (FWIW, I haven't noticed any 503s myself - at least not with anything approaching regularity.)
-
100 or 200 is 'a ton.' 8 certainly isn't, and I regularly open 5-10 at a time to catch up on unreads.
Can we just go straight to "your browsing habits are wrong"?
1.5G of memory, which craps itself with more than 4 (DC) tabs
What. Do web forums have minimum hardware requirements for users now?
-
Can we just go straight to "your browsing habits are wrong"?
Then move swiftly on to "no they aren't, that's a perfectly cromulent way of browsing a multi-page website"? Ok.
What. Do web forums have minimum hardware requirements for users now?
Apparently so. That said, it's not the best box I have at my disposal at work, but I usually only use it for browsing web stuff and a few ssh sessions when I'm doing other stuff on my laptop.
-
What. Do web forums have minimum hardware requirements for users now?
Of course they do. Doubly so if you're on mobile, because anything less than top of the range Android (< 1% of market?) is doing it wrong.
-
Just to confirm, so the main way to repro this is to have a ton of tabs (8+) open? No special behavior other than "tons of tabs?"
I was getting limited just reading a single topic if the posts aren't 10 paragraphs long, as noted in post 49. Bumping it to 200 requests/minute should be sufficient enough to stop the problem reading a single topic with zero other tabs open.
-
Can we just go straight to "your browsing habits are wrong"?
No, you see, Discourse is not compatible with tabbed browsers. You must downgrade to IE6. In fact, I shall go file a bug report with Chromium that they are Doing it Wrong.
-
Chromium
is - contrary to popular opinion - not supposed to be a browser, but rather a tabbed Window Manager for the Internet. http://dev.chromium.org/user-experience
-
In the long term, we think of Chromium as a tabbed window manager or shell for the web rather than a browser application. We avoid putting things into our UI in the same way you would hope that Apple and Microsoft would avoid putting things into the standard window frames of applications on their operating systems.
... not that i disagree with that sentiment but.....
edit actually:
-
I look forward to the new and innovative ways in which pre-existing browser features can be crippled.
Maybe it can break sear-- oh.
What about the back but-- oh.
Uh, favourites? I'm not yet aware of any way that Discourse breaks favourites.
-
Maybe it can break sear-- oh.
What about the back but-- oh.
Uh, favourites? I'm not yet aware of any way that Discourse breaks favourites.
That depends if the history-spunking option has been disabled via plugin.
-
Just checking the default rate limiting template in the image, it is 10 requests in a second and 100 per minute from the same IP. The 100 per minute strikes me as a little low for a default.
This rate limiting doesn't seem to take into account people who browse the forum as I do:
- Open topic page.
- Middle click on topics that I wish to continue/begin reading. (Look, new tabs/windows!)
I often get up to as many as a dozen tabs this way. Looking at the network requests from these tabs, each one appears to initiate a long polling request every 12 seconds. So, just by having a dozen tabs open, I'm already up to 60 requests per minute, and I'm not even reading anything yet! Let alone liking, posting, bookmarking, ...
It should probably be somewhere around 720 if we are going to allow sustained 12 requests per second, or somewhere around 500 if we're going to throttle sustained down a bit.
I'd agree that something like 500 seems a lot more reasonable.
Edit: I now see that @darkmatter has made this point above. With pictures too!
-
Or just not having all the polling crap in the first place for 'metrics' that don't help anyone or achieve anything useful.
The 'reading time' one in particular strikes me as overthought.
-
@codinghorror should include this in his polling logic
var window_focus; $(window).focus(function() { window_focus = true; }) .blur(function() { window_focus = false; }); $(document).one('click',function() { setInterval(function() { $('body').append('has focus? ' + window_focus + '<br>'); }, 1000); });if has focus = false, TURN DOWN THE FUCKING POLLING TO SANE LEVELS
-
for 'metrics' that don't help anyone or achieve anything useful.
The reading time metric is how dicsourse tells whether you've read a post or not, and where to dump you into the middle of the topic at the next time you open it. The last post # read is where the topic will load the next time.
What needs to be changed is 2 things
-
There should be no polling on windows that are not focused - there is no point polling for a UI that isn't showing anything. It can poll for updates when it gets focus instead.
-
The poll should not get auto-triggered by the timings update every time you scroll.... yes that is currently how the notifications thing realizes that you've read the thing it was trying to notify you about, but jesus, that just doubles the network traffic when you could.. you know.. RETURN THE DATA FOR POLLING RESPONSE IN THE TIMINGS POST RESPONSE (which is currently just an empty response). That halves the network traffic necessary right there.
-
-
When you scroll, for some reason, it polls basically between every single timings post.
Now I realize that the purpose of this is likely to cause an immediate update to the notifications widget to let it know that you've read the post it was trying to notify you about. This really should be combined into the response from the Timings post, which would cut the necessary network traffic in half!
-
Ah, but the reading time metric is actually 'used' by Discourse to judge the average reading time of a topic in itself in the opening post's area as if this is in any way useful.
Cut back on the timing metric - cuts bandwidth - then you only need to really flag read status when you request more posts. And you get to avoid all the per-post tracking of what's been read simply to 'furthest point in the topic'.
-
Point still stands. They are doing it wrong.
-
You know, I don't know why windows that don't have focus do not reduce polling rate. That is a good idea. I will ask Sam.
Current defaults FYI are:
- long polling interval: 15s
- polling interval: 3s
- anon polling interval: 30s
Maybe we should add
- unfocused polling interval: 30s?
-
unfocused polling interval: 30s?
Why poll at all? Add the request to the window's focus event and only hit your server when you actually needed data.
Even at 30s polling intervals, people that open 10 background tabs and then get interrupted and leave their computer without remembering to close their browsers are still generating 20 useless requests a minute.
-
I would say no UI poll updates EXCEPT for the unread post indicator (that you use in your tab tip - and possibly the blue bar) - and even then, at maximum 1 update per minute or so, and that's honestly still pretty aggressive.
Discourse chews through mobile battery life like a fucking video because of the amount of idle work it does and the work while focused. One of Discourse's primary goals for the next major revision is reducing the sheer amount of network traffic and logical data. You know that guy who was complaining about discourse issuing millions of DNS lookups to his site on meta.d? That got brushed off. But it outlines a fundamental issue with discourse:
Put simply, it's doing too much, too frequently. People should be able to load +- 50 posts, and not have constant updates. You should send a post id, and a timestamp for changing data lookups, and issue a second request for more data. It might sound like you're asking for more data than otherwise, but once posted (and outside of ninja edit range) posts rarely change in a significant way so you save a ton of bandwidth and resources on payload data.
After about an hour, it is EXTREMELY unlikely anything changes with those posts.
You could get huge lifts in discourse performance by having a job that slowly writes out static HTML for changed topics (even if it's just a JSON feed of all the data) - you would need to be careful with privacy settings, flag settings, etc. But writing the post content out as a static file would make your scrolling requirements damn near instant, and you could load thousands of posts in the same time it takes to load 20 right now. You also wouldn't have the 'Scroll through profile post history crashes the browser' bug.
-
Actually turns out most of this is due to bugs in Discourse.
This may come as a shock to ... none of you.
Good feedback @matches, I agree there is more we can do here, but the bad behavior here is mostly that bug. You can view github checkins for the details, etc.
-
I'm... wow.
Thank you for your honesty.
-
This may come as a shock to ... none of you
a little bit actually, in a pleasant way.
thanks for your honesty and the feedback, it is much appreciated.
-
I'm shocked to the contrary.
-
Now I realize that the purpose of this is likely to cause an immediate update to the notifications widget to let it know that you've read the post it was trying to notify you about. This really should be combined into the response from the Timings post, which would cut the necessary network traffic in half!
Is anyone else reminded of performing database queries on the MouseMove event?
-
I'm out of likes for the next 20 hours (thanks to that thread), so have a +1 instead.
-
Assuming there have been deploys of the latest version here recently, the 503s should be more or less eliminated, even with 10+ tabs open.
-
Assuming there have been deploys of the latest version here recently
Generally I do it between 0800-1000UTC weekdays, Sometimes weekends and other times. I log them here so everyone else here knows when it's happened, and which version we're on.
-
Can we just go straight to "your browsing habits are wrong"?
Then move swiftly on to "no they aren't, that's a perfectly cromulent way of browsing a multi-page website"? Ok.
Multiple pages are a barrier to reading. The entire Web should clearly be a single infiniscroll page.
-
Ick. You can't implement infinite scroll for webcomics! Webcomics move to the side, not up and down.
Instead, they use 'infinite' → technology. Example: http://www.avasdemon.com/pages.php#14
-
You can't implement infinite scroll for webcomics! Webcomics move to the side, not up and down.
oh yee of little faith.
you want infiniscroling webcomics? i can make infiniscroling webcomic!
oh wait. someone already did!
-
I'm finding myself wanting to scroll down to the start of the storyline and scroll up. This is not the most convenient way to read; it could've been done better.
-
-
Are we angling for that new necro badge?
-
You mean the one that hasn't been implemented yet?
-
Yes. Perhaps a head start?
-
Anyone seen 503s in the last 2 days?
-
not that i've noticed.
@zoidberg are you still here or have you gone awol because of a 503?
Hmm.... He's been AWOL for at least 20 hours. Log file rolled over at that point.
damn i need to get a better watchdog in there.
-
I don't recall anything annoying me enough to open up dev tools.
-
None that I'm aware of
-
BTW: if you want to set up a sockbot instance of your own it'll now try to PM you whenever it gets a 503 polling /notifications.
of course the PM might not get through but that is something. I plan to make it be able to email them too for more reliable delivery
Dilbert.com
