Peas, jest here mii out awn these won

accalia

Sam, have you ever addressed the concerns of people who point out that plenty of forums legitimately have threads like that?

and regardless of whether those threads count as legitimate or not, why were we in a situation not too long ago where polling /t/1000/1.json once every fifteen seconds causing the site to experience constant interruptions of service, from one server loading one JSON page four times a minute?

I'll even admit that the work we're doing with http://servercooties.com is not a legitimate use of the forum resources, but at the same time the only reason that Myself, @Onyx, @RaceProUK and @Yamikuronue put that site together in the first place was to monitor the service interruptions of Discourse so that users could see if their problems with the site were issues with their connection or with the site itself.

I would argue that the existence of ServerCooties.com is pretty damning evidence that you have a critically bad problem with Discourse. It is certainly at least partially based in technical reasons, but it's also a PR issue.

I'm glad to see you acknowledging the performance issues, at least in part, that's a good sign to me. It's a start.

Yamikuronue

@sam said:

The whole liking thing

Is it the likes that are breaking Discourse? I was under the impression it was the number of posts.

sam

I already fixed a bunch of issues, but the "act" of doing the liking and "catching up on likes" programmatically is hurting, quite a bit.

Yamikuronue

Forum-goers, are we willing to cool it with the liking in order to keep the thread? Maybe we can compromise here, ease off to preserve performance if the team can honestly give us some timeframe for a fix.

sam

I wish I could give an exact timeframe, I can not, I am however looking at this again today to see if I can place some hard limits to protect ourselves.

RaceProUK

We already reduced the number of auto-likers; what more do you want?

In all seriousness, laying off the liking in t/1000 kinda defeats the purpose of it in a way. But it's up to others if they lighten up or not; currently, I'm only manually liking new posts. All my auto-likers are off, and they're not binging.

@sam said:

I am however looking at this again today to see if I can place some hard limits to protect ourselves.

…what sort of limits?

nightware

The coincidental timing that gave that thread the t/1000 ID...

The high post count and the likes are not the only oddities with that thread. It's been used for lots of experimentation. For example, the first 750 posts have multiple incoming links from a other threads, such as:

http://what.thedailywtf.com/t/how-many-one-boxes-can-be-one-boxed-lets-include-likes-topic-warn-page-is-1mb/9008
http://what.thedailywtf.com/t/glados-runs-code/47472/67?u=nightware

Meh, probably unrelated, but just saying, that thread....

FrostCat

@sam said:

I already fixed a bunch of issues, but the "act" of doing the liking and "catching up on likes" programmatically is hurting, quite a bit.

I really hope you can fix this so it doesn't cause so much CPU or whatever usage, rather than by rate-limiting stuff.

FrostCat

@sam said:

But... the reality is that out of our 200 customers the largest topic, globally is 5688 posts. So its not exactly my top priority at the moment.

That's actually a bit surprising to me, that it's that low, knowing some of the sites that use Discourse.

accalia

well those are just the paying customers. are the ones you are thinking of paying or self hosted?

RaceProUK

Here's an idea…

The worst of the performance issues happen when people are catching up on Likes, right? It's a perfect storm of heavyweight topic loading and on-demand badge queries; IIUC, the Nice/Good/Great Post badge queries run on every Like. So why not have some sort of background queue to run those at a rate that doesn't overload the server? Saving the task to the queue should be quick (one row in one table quick), and at quiet times, like right now, the background task can catch up. That way, the load is spread out over a longer time, reducing peak load, and making things smoother.

FrostCat

@accalia said:

well those are just the paying customers. are the ones you are thinking of paying or self hosted?

I assume BoingBoing, for example, is paying, and I assume (based on no evidence) they would occasionally produce megathreads.

accalia

@FrostCat said:

would occasionally produce megathreads.

i'd tend to think the opposite really they seem to aggressively push the new stuff so the older threads die fairly quickly.

RaceProUK

Longest (public) thread on BoingBoing is 5690 posts

…and based on what @sam posted earlier, that confirms BoingBoing as a paying Discourse customer

FrostCat

Given that you don't get badges in realtime anyway this seems like a sensible idea to me.

FrostCat

@accalia said:

i'd tend to think the opposite really they seem to aggressively push the new stuff so the older threads die fairly quickly.

You would think, but since @RaceProUK says in the next post they're the ones with the biggest topic...
My thought was that since they have a big audience, people might try to keep posting in a popular comment thread every once in a while even if it goes off the front page so that you'd mostly have shorter threads, but sometimes you'd get a really big one.

accalia

@FrostCat said:

You would think

i've often been very very wrong too. ;-)

FrostCat

@accalia said:

i've often been very very wrong too.

That's as may be, but I wasn't using the personal you there.

RaceProUK

Funnily enough, that thread I mentioned… appears to be a forum game

Mar 27, 2015

Did you ever want to play questions?

Continuing the discussion from Have you ever heard of: Who else is playing? @Mindysan33 @AcerPlatanoides @japhroaig @funruly @daneel @popobawa4u @miasm @SteampunkBanana &c #Could you please ask a question?

Groaner

@Onyx said:

Out of personal fucking principle of not letting shit left broken because someone's pantyhose gets in a bunch when someone mentions 42k posts in a single thread.

I wonder what happens with the 65536th mention.

Or the 2147483648th. Because signed limits seem to be all the rage these days.

darkmatter

@sam said:

approx 176ms of CPU work on the server grabbing every post number in the topic (discounting cheap db work)

ouch... 176ms of CPU work - is that by 100% CPU usage meter or just that it took 176ms but probably only used about 1% CPU?

@RaceProUK said:

So why not have some sort of background queue to run those at a rate that doesn't overload the server?

I like - no more goddamn insta-nice post s within 5 seconds of posting is not a bad thing.... even when it's a legit nice post in a non-gamed topic, it's not like i care to see immediately after the 10th or whatever-th person likes it that i've been loved. ~~except as I understand it, those are the pre-built ones that you can't turn off. That's why we have custom SQL ones that mean something, and they ignore t/1000.~~ oh wait you're telling sam to background queue the built-in ones, that makes perfect sense, nevermind.

Mikael_Svahnberg

@sam said:

the topic was there to "prove" you can break Discourse.

You "broke" Discourse hence you "win", if it was me I would just archive and close the topic.

That /t/1000 was created to prove this may be the case, but interestingly enough it keeps on proving that Discourse can be broken in new and innovative ways. And that's sort of a key here.

@RaceProUK said:

Here's an idea…

The worst of the performance issues happen when people are catching up on Likes, right? It's a perfect storm of heavyweight topic loading and on-demand badge queries; IIUC, the Nice/Good/Great Post badge queries run on every Like. So why not have some sort of background queue to run those at a rate that doesn't overload the server? Saving the task to the queue should be quick (one row in one table quick), and at quiet times, like right now, the background task can catch up. That way, the load is spread out over a longer time, reducing peak load, and making things smoother.

Cloud Architecture Patterns, Chapter 3.

sam

I did spend most of today looking at general perf:

May 13, 2015

PERF: Active Record #pluck is twice faster · discourse/discourse@51d82fc

PERF: SqlBuilder bypasses AR converters and uses PG directly

and

May 13, 2015

Double request to topic json when navigating to a topic

I am noticing double request for topic json being initiated from “topic” and “fromParams”. The repro is quite simple: Navigate to a topic from topic list Click back Click on topic again and you get 2 json requests to topic just to load it. @eviltrout can you have a look at this

Which will be fixed shortly.

Keep in mind I am focusing on areas that generally improve perf first cause everyone wins that way.

Maciejasjmj

@NedFodder said:

They've already told us that 42K posts is Doing It Wrong™

So we just wait until a big customer hits that limit. Say, BoingBoing.

I approve (somewhat) of locking it temporarily for debugging purposes, but a 42K post thread is not Doing it Wrong, it's using the software as it's intended to. If your software breaks when used as intended, your software is broken, and you can't blame the client.

And every other forum software works with megathreads. Fuck, probably even CS would handle it if it didn't have too many tags. If having a big thread grinds the forum to a halt, once again, the forum is broken. Not the users.

@Onyx said:

I made the original raw button.

Oy, your original raw button was a link to /raw/! I demand my kudos!

@sam said:

cause active record is just slow that way

Stop using it.

@sam said:

You "broke" Discourse hence you "win"

And the prize is... that we have a broken forum software? Look, you've been in beta when it started, so we tested stuff and expected it to be fixed. It was okay, it was a beta, you didn't have too many real forums back then, things would break.

Now you're way past 1.0, and your forum software can't handle a big thread. You know, the thing that happens when many people use your software. It's no longer okay. It means you shouldn't have gone out of beta without fixing this, because sooner or later, other forums will also start hitting this kind of numbers.

And frankly, generating a big thread and looking at the performance is something you should have done before even releasing the software to anyone. There's no excuse. Any other forum software can handle threads of - probably - around INT_MAX posts, and you claim to be better.

wft

@blakeyrat said:

forum software that isn't the shit.

Does not exist.

Whatever you may think about the piece of fucking shit the Community Server was is due to the imprinting effect.

Onyx

@Maciejasjmj said:

Oy, your original raw button was a link to /raw/! I demand my kudos!

It wasn't a link but that's the way it was pulled, yes, "API" limitations. Kudos for any improvements still granted, but I don't want no lies here.

@Maciejasjmj said:

Stop using it.

+1. This seems to be the primary bottleneck. Offload at least some of this stuff where it belongs: the DB. Also, I know MySQL is a pain, I really do, but why even keep it? Am I wrong in claiming you support it btw? I don't see a justification for it, Discourse can't run on a cheap PHP shared host anyway. Just give up on the damned thing and use a single RDBMS. I root for Postgres, personally.

Also, I would love if @sam could confirm or refute some of my assumptions about how this whole thing works. I may be wrong on multiple accounts and I'd love to be corrected where necessary.

I would also like a confirmation about likes themselves being the problem. I know they were in the past, and I seem to remember something was done about it. Am I wrong there? Is it the badge queries after all?

wft

@Onyx said:

Now, since it takes you a while to read those 20 posts, and everything is updated live, there might be posts moved, deleted or whatever in that time. So, the most sensible thing? Do the whole thing over once you load the next batch, of course!

Reminds me of an old article where heavy computing was all in onMouseOver handler.

@Onyx said:

MyISAM (which doesn't support foreign keys) is faster than InnoDB, IIRC.

No longer an universal truth.

@Onyx said:

No stored procs, no triggers, no foreign keys.

KILL IT WITH FIRE

@Onyx said:

So... referential integrity and all that? Yeah, all in Ruby

DID YOU HEAR ME I TOLD YOU TO KILL IT WITH FIRE!

Onyx

@wft said:

No longer an universal truth.

I believe it still is at face value? As in, you just make a simple "by the books normalized" DB and just start using it without specific tweaks. Given the current schema and the fact that it needs to support Postgres as well, I am unsure as to how much tweaking is even done.

Maciejasjmj

@Onyx said:

It wasn't a link but that's the way it was pulled, yes, "API" limitations.

Tsk, tsk.

actionArea.prepend('<button title="view raw post" class="tdwtf-view-raw" onclick="window.open(\'http://what.thedailywtf.com/raw/' + topicID + '/' + postID + '\', \'_blank\');"><i class="fa fa-code"></i>#' + topicID + ':' + postID + '</button>');

Well, technically not a link, but it still wasn't until my post that the raw actually started showing in the post area instead of a new tab.

@Onyx said:

Just give up on the damned thing and use a single RDBMS. I root for Postgres, personally.

It makes even less sense if you consider that Discourse generally comes in a prepackaged Docker container. There's pretty much zero point in handling multiple RDBMSes.

Onyx

@Maciejasjmj said:

Well, technically not a link, but it still wasn't until my post that the raw actually started showing in the post area instead of a new tab.

Curse your sudden but inevitable use of historical data!

I stand corrected.

sam

Here is 120ms out of 400ms it took to display front page for @RaceProUK

there is much pain going on

May 13, 2015

PERF: add missing index on post actions · discourse/discourse@8362993

A platform for community discussion. Free, open, simple. - PERF: add missing index on post actions · discourse/discourse@8362993

boomzilla

So part of this is simply the DB scaling up and uncovering new slow spots? Comparing about pages here and at BoingBoing...

All time:
Topics: 10K / 21K
Posts: 412K / 460K
Likes 1.4M / 739K

We have fewer users but overall we seem more active right now:

30 days:
Topics: 381 / 1.1K
Posts: 32K / 28K
Likes: 144K / 53K

sam

@boomzilla said:

So part of this is simply the DB scaling up and uncovering new slow spots?

Definately, a big part, also the import added some more wood to the fire... I have this running locally and uncovering lots of stuff.

accalia

@sam said:

Definately, a big part, also the import added some more wood to the fire...

Speaking of performance trying to load anything under /user/accalia is taking about five to ten seconds usually to render, sometimes quite a bit more...

not noticing the slowdown visiting other people's profiles though.

anonymous234

My vote is, as usual, to keep the likes thread but without the likes. Turn it into the random things thread.

And delete the existing likes for good measure.

boomzilla

It's very slow to load mine. Yours seemed a little faster for me. Still pretty slow. ~9s vs ~5s.

RaceProUK

Mine's quicker to load, but still pretty slow. Don't have an exact timing, but it's around 6-7 seconds.

Weirdly, it's about that for @boomzilla's and @accalia's too.

sam

Executing action: index
T+23.7 ms
Reader
8301.6 ms
lib/freedom_patches/active_record_base.rb:7:in `exec_sql'
lib/sql_builder.rb:67:in `exec'
lib/sql_builder.rb:94:in `map_exec'
app/models/user_action.rb:186:in `stream'
app/controllers/user_actions_controller.rb:24:in `index'
lib/middleware/anonymous_cache.rb:123:in `call'
lib/middleware/request_tracker.rb:70:in `call'
lib/scheduler/defer.rb:85:in `process_client'
lib/middleware/unicorn_oobgc.rb:95:in `process_client'
      SELECT
        a.id,
        t.title, a.action_type, a.created_at, t.id topic_id,
        t.closed AS topic_closed, t.archived AS topic_archived,
        a.user_id AS target_user_id, au.name AS target_name, au.username AS target_username,
        coalesce(p.post_number, 1) post_number, p.id as post_id,
        p.reply_to_post_number,
        pu.username, pu.name, pu.id user_id,
        pu.uploaded_avatar_id,
        u.username acting_username, u.name acting_name, u.id acting_user_id,
        u.uploaded_avatar_id acting_uploaded_avatar_id,
        coalesce(p.cooked, p2.cooked) cooked,
        CASE WHEN coalesce(p.deleted_at, p2.deleted_at, t.deleted_at) IS NULL THEN false ELSE true END deleted,
        p.hidden,
        p.post_type,
        p.edit_reason,
        t.category_id
      FROM user_actions as a
      JOIN topics t on t.id = a.target_topic_id
      LEFT JOIN posts p on p.id = a.target_post_id
      JOIN posts p2 on p2.topic_id = a.target_topic_id and p2.post_number = 1
      JOIN users u on u.id = a.acting_user_id
      JOIN users pu on pu.id = COALESCE(p.user_id, t.user_id)
      JOIN users au on au.id = a.user_id
      LEFT JOIN categories c on c.id = t.category_id
      WHERE (t.deleted_at is null) AND (a.user_id = 18) AND (a.action_type in (4,5))
      ORDER BY a.created_at desc
      OFFSET 0
      LIMIT 60

8.3 secs the all tab is brutal

aliceif

http://what.thedailywtf.com/user/accalia is inaccessible for me ...

Yamikuronue

Users, plural. http://what.thedailywtf.com/users/accalia

blakeyrat

@sam said:

176 samples are there just doing plucking, cause active record is just slow that way.

"plucking" is a stupid word
if active record is so slow, why the shit are you using it? ORMs are always shitty.

@sam said:

You "broke" Discourse hence you "win", if it was me I would just archive and close the topic.

No. You shipped a broken product. We only showed you it was broken. We're pushing your nose into the shit on the carpet.

This is exactly why we should never close the topic.

@sam said:

So its not exactly my top priority at the moment.

Right; you gotta spend hours doing the crucially important work of dicking around with the CSS to remove stripes from lists.

@sam said:

I am however looking at this again today to see if I can place some hard limits to protect ourselves.

So the great solution is to just arbitrarily declare a max thread size and fuck you if you want longer threads? You know, to "protect" ourselves from... ourselves.

Quality software development here.

@Mikael_Svahnberg said:

but interestingly enough it keeps on proving that Discourse can be broken in new and innovative ways.

No; the breaking was there before the topic existed. The software shipped broken. We're not "breaking" the software, we're showing the Discourse team that they shipped broken software.

It's an important distinction. If we upgraded, say, the version of Ruby and stuff broke, then we'd be breaking Discourse by changing its runtime environment. We didn't do that. We're running exactly the configuration they specify.

@Maciejasjmj said:

Fuck, probably even CS would handle it if it didn't have too many tags.

CS does handle it. Fine. Discourse is significantly shittier than CS in almost every way.

@Maciejasjmj said:

And frankly, generating a big thread and looking at the performance is something you should have done before even releasing the software to anyone. There's no excuse. Any other forum software can handle threads of - probably - around INT_MAX posts, and you claim to be better.

They claim to be better because they've obviously NEVER USED ANY COMPETING SOFTWARE! Like I just said, a lot of shit that Community Server (a JOKE in forum software) got perfectly correct back in 2008, they're getting wrong.

@wft said:

Does not exist.

Possibly; but there are a lot of products out there a lot better than Discourse.

@wft said:

Whatever you may think about the piece of fucking shit the Community Server was is due to the imprinting effect.

The quote button worked. The scroll bar worked.

@sam said:

Here is 120ms out of 400ms it took to display front page for @RaceProUK

If you embarrass Discourse developers enough, they seem to actually maybe start caring a little bit that their software is shit. Good to know I guess?

xaade

@RaceProUK said:

42k posts

Maybe the problem will be fixed when we have 43k posts.

xaade

@sam said:

I would just archive and close the topic

bury it and pretend we don't have a problem.

Why does it matter if there are 42k posts in ONE thread.

Does your software break when there are 42k posts in all threads?

So why the difference?

And I know that's a naive question, but sometimes it takes the 5 year old asking the obvious to put people back on track.

xaade

@blakeyrat said:

So the great solution is to just arbitrarily declare a max thread size and fuck you if you want longer threads? You know, to "protect" ourselves from... ourselves.

It's ok. Let them do it.

It will be the laughing stock of forum software.

Professors in universities will load of Discourse in class, open a thread with n number of replies, try to reply, show the max post count reached message, and tell students this is what you don't do when you write software.

I'm beginning to wonder if this stuff shares code with the insurance "marketplace". They reached a maximum number of applicants and just took a dump on the carpet. Maybe they should limit the number of people that apply to 42k also.

I mean, that seems to be the number of people actually helped by the new laws.

riking

@RaceProUK said:

So why not have some sort of background queue to run those at a rate that doesn't overload the server?

That's already done, liking a post places it into a queue in Redis that is cleared every minute by Sidekiq.

sam

Be sure to read:

http://what.thedailywtf.com/t/docker-upgrades/1929/155?u=sam

There are a pile of optimisations I checked in, additionally I tuned the DB some. CPU is looking a bit better on the box, monitoring slow queries.

swayde

May 13, 2015

PERF: Active Record #pluck is twice faster · discourse/discourse@51d82fc

PERF: SqlBuilder bypasses AR converters and uses PG directly

In freedom_patches/fast_pluck.rb line 8 to 32 you've Committed commented out code (according to syntax highlighting,that is) does that do anything?
What about from line 86 to 108? Is that just a result of a test, or does it do some magic in ruby?

sam

Just a pointer to others that want to rip off this freedom patch

abarker

@boomzilla said:

It's very slow to load mine. Yours seemed a little faster for me. Still pretty slow. ~9s vs ~5s.

Mine takes ~11s compared to ~10s on yours and ~7s on @accalia's.

I'm guessing that there's some slow counting invloved somewhere.

RaceProUK

s?