Boss, would you mind finding another topic to shit all over and go shit on it
This is my topic
Boss, would you mind finding another topic to shit all over and go shit on it
This is my topic
One of these devs is not like the other one, one of these devs does not belong
If we start pitting me against my co-founder here into some sort of trolly cat-fight I am going to have to take a break.
I am using this topic to give the community updates about what I am doing to help cut out the very horrible experience people are having here with forum reliability and performance.
I do this because we truly appreciate the use/abuse Discourse gets here and want to fix bugs. We need to be careful with our time, this is not a paid gig for us and we are not deployed on Discourse infrastructure. I do this cause I appreciate the forum, but any @sam vs @codinghorror shit needs to die.
t/1000 is taking 1.2 seconds, 630ms of those are in SELECT "posts"."id" FROM "posts" WHERE "posts"."topic_id" = $1 ORDER BY "posts"."sort_order" ASC
. This is way above what I find acceptable but not the only reason we were seeing 500s here. I am still determining all the root causes (as the updates to this topic show)
Wait, it doesn't show you a mention count?
Unfortunately we were counting them with Int64 and ran out of space, so omitted from the numbers above.
see:
Snippets from the report here (after running for about 40 minutes):
Total Requests: 62578 ( MessageBus: 31590 )
Top 30 users by Server Load
Username Duration
-------- --------
VinDuv 552.75
[Anonymous] 548.53
PaulaBean 137.83
HardwareGeek 50.38
aliceif 38.43
...
Top 30 routes by Server Load
Route Duration
----- --------
post_actions/create 583.70
topics/show 514.38
user_avatars/show 91.74
topics/timings 86.24
list/latest 42.07
...
Top 30 urls by Server Load
Url Duration
--- --------
POST /post_actions HTTP/1.1 583.70
POST /topics/timings HTTP/1.1 91.66
GET /session HTTP/1.1 36.00
GET /t/2/last HTTP/1.1 28.06
GET / HTTP/1.1 26.30
POST /posts HTTP/1.1 25.75
GET /notifications HTTP/1.1 15.42
GET /t/category-definition-for-meta/2/99999999 HTTP/1.1 14.29
...
Top 30 not found urls (404s)
Url Count
--- -----
GET /session HTTP/1.1 2010
POST /notifications/reset-new HTTP/1.1 131
GET /rules.abe HTTP/1.1 8
What I learned from this...
Liking on t/1000 is brutal, each like is costing us 1-3 seconds of server time. I will look at tuning this some but really, this game is messing you up big time, @VinDuv's recent like spree cost us about 9 minutes of server time.
We are returning 404s sometimes inappropriately. Will clean that up.
t/2 is being accessed a lot
Will see what happens when I run the report again in say 10 or so hours. Will be very interesting to see the results cc @boomzilla @PJH
I just commented out this line:
thing is ... rebaking all the posts you are quoted in may be interesting in theory each time you change avatars. it needs some severe rate limiting in practice, especially here. CPU was pegged cause there was a 4 way rebake going for all the posts a certain user here who changed avatars made, we will redesign so this can not happen.
LOL
I just tried to create the Worst of the Worst badge and had a 422 error... THANK YOU DISCOURSE
In other news, every new post to t/1000 incurs a 2 second query to figure out the "top posters" in the topic.
Gee, I wonder if it's because he's an incompetent dumbshit who can't find his ass with both hands.
I love you too blakey
I did spend most of today looking at general perf:
and
Which will be fixed shortly.
Keep in mind I am focusing on areas that generally improve perf first cause everyone wins that way.
Yeah, isn't that already it's own punishment, though? Let them decide when they've suffered enough and want to bring back what passes for performance with Discurse. This whole "you're using our software wrong!" bullshit stinks to high heaven, Sam.
Read up, I said I will revisit this, will probably just remove the limit, if you decide to work with an inbox 10000 you are going to pay the price of an extra 400ms delay baked into first visits here.
Nothing is being untracked old stuff is dismissed, but topics remain tracked, anyway I will revisit
still only see 200 new topic
New topics are capped at the site level to 2/5th of max_new_unread_topics, out of the box that is 200.
I will revisit this stuff eventually but the banner is not a solution, just a support nightmare.
Especially when the issue could simply be fixed with a tiny bit of copy. "You have more than X unread items; only the latest X will be listed."
Not that simple, we re-sync the in memory data when we load the unread stuff, so I need to change it so it displays 250+ or something like that, its not a simple change.
The original complaint was that unread can not be trusted.
also, I could probably get inbox one million going here if I spent a couple of weeks on it, but honestly I would prefer spending the time on fixing t/1000
usage of the word "rape"
Jesus Christ, is that called "eye rape: the plugin?"
Why the fuck are we talking about not being able to read a clock while ignoring the anally-raped muppet face in the room!
Hey guess what? I don't give a flying fuck about learning markdown. Also you're fucking wrong, look up a line. Now it's one of those rape-victim faces with a semi-colon surgically-attached to its ear.
So because of past poor design decisions, we have to continue serving past poor design decisions by destroying ostensibly useful data?
How many unread topics do you have?
By "self corrects" you mean "deletes information relevant to the user without their knowledge or consent", then... yes.
If you are going for an inbox ONE MILLION and objecting that we do not support that ...
Sorry.
Or if you really want to... let me know how to rewrite this without a limit so it does not rape us.
WITH x AS (
SELECT u.id AS user_id,
topics.id AS topic_id,
topics.created_at,
highest_post_number,
last_read_post_number,
c.id AS category_id,
tu.notification_level
FROM topics
JOIN users u on u.id = 576
JOIN user_stats AS us ON us.user_id = u.id
JOIN categories c ON c.id = topics.category_id
LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id
WHERE u.id = 576 AND
topics.archetype <> 'private_message' AND
(("topics"."deleted_at" IS NULL AND tu.last_read_post_number < topics.highest_post_number AND COALESCE(tu.notification_level, 1) >= 2) OR ("topics"."deleted_at" IS NULL AND topics.created_at >= GREATEST(CASE
WHEN COALESCE(u.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
WHEN COALESCE(u.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(u.previous_visit_at,u.created_at)
ELSE ('2015-09-09 21:12:17.314182'::timestamp - INTERVAL '1 MINUTE' * COALESCE(u.new_topic_duration_minutes, 2880))
END, us.new_since) AND tu.last_read_post_number IS NULL AND COALESCE(tu.notification_level, 2) >= 2)) AND
(topics.visible OR u.admin OR u.moderator) AND
topics.deleted_at IS NULL AND
( NOT c.read_restricted OR u.admin OR category_id IN (
SELECT c2.id FROM categories c2
JOIN category_groups cg ON cg.category_id = c2.id
JOIN group_users gu ON gu.user_id = 576 AND cg.group_id = gu.group_id
WHERE c2.read_restricted )
)
AND NOT EXISTS( SELECT 1 FROM category_users cu
WHERE last_read_post_number IS NULL AND
cu.user_id = 576 AND
cu.category_id = topics.category_id AND
cu.notification_level = 0)
ORDER BY topics.bumped_at DESC ) SELECT * FROM x LIMIT 500
Limit can be set by admins ... but at 500 it's already taking 150ms here for my account, so yeah we could bump it up to 100000 in site settings here and people with one million unread can wait an hour to get the front page.
So win.
You can read some of the history here https://meta.discourse.org/t/new-unread-counts-missing-when-too-many-tracked-topics/32922
The banner should be gone though on reload, on the version you are running