Server cooties are being FIXED, NOT swapped

[sam]

materialized views in postgres kind of suck cause you need to keep issuing refreshes to them, but I guess it could be done, if a plugin was added that does that.

Nov 8, 2018

Materialized Views

[Weng]

Every time discosql gets posted, it twigs my SQL Server code smell. I have no idea how closely pg matches SQL Server on these concerns, but....

Too much nesting. Build temp tables and join on those, and trade where-in statements for joins (yeah the query optimizer might do the right thing anyway, but it's an asshole and will change its mind to your detriment in future)

Paranoid locking of tables that are read will cause excess blocking. If you're not going to fuck with the table and it isn't timing critical, hint away the locks or run at snapshot isolation.

Oh, and that GUID string match in the middle. What. The. Fuck. On every post by every admin. Clearly that string means something, but that something should be a fucking property of a post, not a goddamn magic string buried in the body.

[Polygeekery]

@Weng said:

Oh, and that GUID string match in the middle. What. The. Fuck. On every post by every admin. Clearly that string means something, but that something should be a fucking property of a post, not a goddamn magic string buried in the body.

The GUID is for excluding threads from badges. It always seemed like a silly (yet easy to implement) way to go about that. It simply has to add a shit load of overhead to the badge queries.

[HardwareGeek]

@Weng said:

that GUID string match in the middle. What. The. Fuck.

@PJH is no doubt asleep, but I'm reasonably sure that's the magic GUID that, when posted to a topic by an admin (or mod?), excludes it from being counted toward automatic badges. So, by posting it here, @sam just excluded this topic.

[jaming]

Yeah, I was thinking that a more efficient way might be to have a PM post with the topic ids to exclude. Killing that LIKE should be a marked improvement, I'd think.

[Polygeekery]

Yeah, the alternative would be editing the SQL every time you want to exclude another topic, or just excluding the likes topic and leaving everything else alone. I am for just excluding the likes topic.

Or, adding functionality to Discourse, which would be fraught with peril...

[Weng]

Assuming there's an index on userid and raw, could turn it from a text search to an index scan by the expedient of requiring it to be at the very start of the raw (drop the leading %).

Or add a table and manually update it with topic ids. Or knock up a PM conversation or secret forum topic for admins where the raw of each post is a topic id to be excluded and join against posts in that topic. Or any of a gazillion shitty hacks that aren't a magic post with a magic string in an arbitrary position.

[HardwareGeek]

@Weng said:

Or add a table and manually update it with topic ids.

I may be wrong, but I'm under the impression @PJH doesn't have the access to do that.

[sam]

@HardwareGeek said:

So, by posting it here, @sam just excluded this topic.

not really I sanitized it ...

[Monarch]

This post is deleted!

[sam]

@Weng said:

Assuming there's an index on userid and raw, could turn it from a text search to an index scan by the expedient of requiring it to be at the very start of the raw (drop the leading %).

we already have a full text index on cooked which could possibly be used.

[sam]

@Weng said:

discosql

Not fair, I did not write that SQL, Discourse admins can define badges and supply SQL, which is what happened here.

Temp tables would need a plugin to populate them.

[sam]

Logs for last few hours:

sam@ubuntu tmp % ruby monitor.rb 
2015-02-20 13:53:39 +1100: 60 successful reqs
2015-02-20 13:54:51 +1100: slow request (1256 ms)
2015-02-20 13:55:14 +1100: 60 successful reqs
2015-02-20 13:56:49 +1100: 60 successful reqs
2015-02-20 13:58:24 +1100: 60 successful reqs
2015-02-20 13:59:59 +1100: 60 successful reqs
2015-02-20 14:01:36 +1100: 60 successful reqs
2015-02-20 14:03:13 +1100: 60 successful reqs
2015-02-20 14:04:48 +1100: 60 successful reqs
2015-02-20 14:06:23 +1100: 60 successful reqs
2015-02-20 14:07:59 +1100: 60 successful reqs
2015-02-20 14:09:34 +1100: 60 successful reqs
2015-02-20 14:11:09 +1100: 60 successful reqs
2015-02-20 14:12:44 +1100: 60 successful reqs
2015-02-20 14:14:20 +1100: 60 successful reqs
2015-02-20 14:15:55 +1100: 60 successful reqs
2015-02-20 14:17:30 +1100: 60 successful reqs
2015-02-20 14:17:42 +1100: slow request (1447 ms)
2015-02-20 14:19:06 +1100: 60 successful reqs
2015-02-20 14:20:41 +1100: 60 successful reqs
2015-02-20 14:22:16 +1100: 60 successful reqs
2015-02-20 14:23:10 +1100: slow request (5741 ms)
2015-02-20 14:23:57 +1100: 60 successful reqs
2015-02-20 14:25:10 +1100: slow request (1910 ms)
2015-02-20 14:25:34 +1100: 60 successful reqs
2015-02-20 14:27:09 +1100: 60 successful reqs
2015-02-20 14:28:44 +1100: 60 successful reqs
2015-02-20 14:30:19 +1100: 60 successful reqs
2015-02-20 14:31:54 +1100: 60 successful reqs
2015-02-20 14:33:20 +1100: slow request (1055 ms)
2015-02-20 14:33:31 +1100: 60 successful reqs
2015-02-20 14:34:42 +1100: slow request (1278 ms)
2015-02-20 14:35:08 +1100: 60 successful reqs
2015-02-20 14:36:43 +1100: 60 successful reqs
2015-02-20 14:38:18 +1100: 60 successful reqs
2015-02-20 14:39:53 +1100: 60 successful reqs
2015-02-20 14:41:27 +1100: 60 successful reqs
2015-02-20 14:43:02 +1100: 60 successful reqs
2015-02-20 14:44:24 +1100: slow request (1282 ms)
2015-02-20 14:44:38 +1100: 60 successful reqs
2015-02-20 14:46:13 +1100: 60 successful reqs
2015-02-20 14:47:21 +1100: slow request (1449 ms)
2015-02-20 14:47:49 +1100: 60 successful reqs
2015-02-20 14:49:24 +1100: 60 successful reqs
2015-02-20 14:50:59 +1100: 60 successful reqs
2015-02-20 14:52:34 +1100: 60 successful reqs
2015-02-20 14:54:08 +1100: 60 successful reqs
2015-02-20 14:55:44 +1100: 60 successful reqs
2015-02-20 14:57:19 +1100: 60 successful reqs
2015-02-20 14:58:53 +1100: 60 successful reqs
2015-02-20 15:00:29 +1100: 60 successful reqs
2015-02-20 15:02:05 +1100: 60 successful reqs
2015-02-20 15:03:30 +1100: slow request (1889 ms)
2015-02-20 15:03:43 +1100: 60 successful reqs
2015-02-20 15:03:49 +1100: slow request (1911 ms)
2015-02-20 15:04:03 +1100: slow request (1025 ms)
2015-02-20 15:04:20 +1100: slow request (1036 ms)
2015-02-20 15:04:24 +1100: slow request (1016 ms)
2015-02-20 15:04:31 +1100: slow request (1782 ms)
2015-02-20 15:04:46 +1100: slow request (5972 ms)
2015-02-20 15:05:07 +1100: slow request (1077 ms)
2015-02-20 15:05:23 +1100: slow request (1094 ms)
2015-02-20 15:05:25 +1100: slow request (1242 ms)
2015-02-20 15:05:41 +1100: slow request (1012 ms)
2015-02-20 15:05:47 +1100: 60 successful reqs
2015-02-20 15:05:53 +1100: slow request (1014 ms)
2015-02-20 15:06:10 +1100: slow request (1005 ms)
2015-02-20 15:06:14 +1100: slow request (1745 ms)
2015-02-20 15:06:18 +1100: slow request (1015 ms)
2015-02-20 15:06:37 +1100: slow request (1904 ms)
2015-02-20 15:06:49 +1100: slow request (3110 ms)
2015-02-20 15:06:59 +1100: slow request (5808 ms)
2015-02-20 15:07:19 +1100: slow request (1954 ms)
2015-02-20 15:07:21 +1100: slow request (1014 ms)
2015-02-20 15:07:27 +1100: slow request (1023 ms)
2015-02-20 15:07:33 +1100: slow request (1033 ms)
2015-02-20 15:07:50 +1100: 60 successful reqs
2015-02-20 15:09:25 +1100: 60 successful reqs
2015-02-20 15:10:59 +1100: 60 successful reqs
2015-02-20 15:11:14 +1100: slow request (1017 ms)
2015-02-20 15:12:09 +1100: slow request (1027 ms)
2015-02-20 15:12:35 +1100: 60 successful reqs
2015-02-20 15:14:13 +1100: 60 successful reqs
2015-02-20 15:15:49 +1100: 60 successful reqs
2015-02-20 15:17:26 +1100: 60 successful reqs
2015-02-20 15:19:03 +1100: 60 successful reqs
2015-02-20 15:20:41 +1100: 60 successful reqs
2015-02-20 15:21:22 +1100: slow request (1587 ms)
2015-02-20 15:22:20 +1100: 60 successful reqs
2015-02-20 15:23:56 +1100: 60 successful reqs
2015-02-20 15:24:44 +1100: slow request (1089 ms)
2015-02-20 15:25:33 +1100: 60 successful reqs
2015-02-20 15:27:10 +1100: 60 successful reqs
2015-02-20 15:28:47 +1100: 60 successful reqs
2015-02-20 15:30:24 +1100: 60 successful reqs
2015-02-20 15:32:02 +1100: 60 successful reqs
2015-02-20 15:32:29 +1100: slow request (1941 ms)
2015-02-20 15:33:40 +1100: 60 successful reqs
2015-02-20 15:35:17 +1100: 60 successful reqs
2015-02-20 15:36:54 +1100: 60 successful reqs
2015-02-20 15:38:04 +1100: slow request (1034 ms)
2015-02-20 15:38:32 +1100: 60 successful reqs
2015-02-20 15:40:08 +1100: 60 successful reqs
2015-02-20 15:41:45 +1100: 60 successful reqs
2015-02-20 15:43:13 +1100: slow request (1216 ms)
2015-02-20 15:43:22 +1100: 60 successful reqs
2015-02-20 15:44:24 +1100: slow request (1095 ms)
2015-02-20 15:44:59 +1100: 60 successful reqs
2015-02-20 15:46:13 +1100: slow request (1012 ms)
2015-02-20 15:46:36 +1100: 60 successful reqs
2015-02-20 15:48:13 +1100: 60 successful reqs
2015-02-20 15:49:14 +1100: slow request (1738 ms)
2015-02-20 15:49:52 +1100: 60 successful reqs
2015-02-20 15:50:57 +1100: slow request (1789 ms)
2015-02-20 15:51:32 +1100: 60 successful reqs
2015-02-20 15:52:26 +1100: slow request (5551 ms)
2015-02-20 15:53:08 +1100: slow request (1043 ms)
2015-02-20 15:53:10 +1100: slow request (1047 ms)
2015-02-20 15:53:17 +1100: 60 successful reqs
2015-02-20 15:53:59 +1100: slow request (1323 ms)
2015-02-20 15:54:01 +1100: slow request (1259 ms)
2015-02-20 15:54:03 +1100: slow request (1408 ms)
2015-02-20 15:54:58 +1100: 60 successful reqs
2015-02-20 15:56:37 +1100: 60 successful reqs
2015-02-20 15:57:40 +1100: slow request (5535 ms)
2015-02-20 15:58:19 +1100: 60 successful reqs
2015-02-20 15:59:57 +1100: 60 successful reqs
2015-02-20 16:01:32 +1100: 60 successful reqs
2015-02-20 16:03:10 +1100: slow request (1794 ms)
2015-02-20 16:03:10 +1100: 60 successful reqs
2015-02-20 16:04:22 +1100: slow request (1410 ms)
2015-02-20 16:04:47 +1100: 60 successful reqs
2015-02-20 16:06:23 +1100: 60 successful reqs
2015-02-20 16:06:56 +1100: slow request (1272 ms)
2015-02-20 16:07:10 +1100: slow request (1545 ms)
2015-02-20 16:08:00 +1100: 60 successful reqs
2015-02-20 16:08:40 +1100: slow request (1172 ms)
2015-02-20 16:09:19 +1100: slow request (5525 ms)
2015-02-20 16:09:41 +1100: 60 successful reqs
2015-02-20 16:11:20 +1100: 60 successful reqs
2015-02-20 16:12:57 +1100: 60 successful reqs
2015-02-20 16:14:01 +1100: slow request (1036 ms)
2015-02-20 16:14:14 +1100: slow request (1015 ms)
2015-02-20 16:14:19 +1100: slow request (1059 ms)
2015-02-20 16:14:33 +1100: slow request (1963 ms)
2015-02-20 16:14:42 +1100: 60 successful reqs
2015-02-20 16:15:18 +1100: slow request (1151 ms)
2015-02-20 16:16:23 +1100: 60 successful reqs
2015-02-20 16:18:01 +1100: 60 successful reqs
2015-02-20 16:19:38 +1100: 60 successful reqs
2015-02-20 16:21:15 +1100: 60 successful reqs
2015-02-20 16:22:52 +1100: 60 successful reqs
2015-02-20 16:24:29 +1100: 60 successful reqs
2015-02-20 16:25:10 +1100: slow request (1588 ms)
2015-02-20 16:25:13 +1100: slow request (1612 ms)
2015-02-20 16:26:09 +1100: 60 successful reqs
2015-02-20 16:27:05 +1100: slow request (1440 ms)
2015-02-20 16:27:25 +1100: slow request (1190 ms)
2015-02-20 16:27:32 +1100: slow request (1301 ms)
2015-02-20 16:27:34 +1100: slow request (1090 ms)
2015-02-20 16:27:49 +1100: 60 successful reqs
2015-02-20 16:28:36 +1100: slow request (1107 ms)
2015-02-20 16:29:25 +1100: 60 successful reqs
2015-02-20 16:30:18 +1100: slow request (1014 ms)
2015-02-20 16:31:02 +1100: 60 successful reqs
2015-02-20 16:32:25 +1100: slow request (1088 ms)
2015-02-20 16:32:45 +1100: 60 successful reqs
2015-02-20 16:33:32 +1100: slow request (1929 ms)
2015-02-20 16:34:33 +1100: 60 successful reqs
2015-02-20 16:35:29 +1100: slow request (1025 ms)
2015-02-20 16:36:19 +1100: 60 successful reqs
2015-02-20 16:37:53 +1100: 60 successful reqs
2015-02-20 16:39:28 +1100: 60 successful reqs
2015-02-20 16:41:04 +1100: 60 successful reqs
2015-02-20 16:42:39 +1100: 60 successful reqs
2015-02-20 16:42:49 +1100: slow request (1004 ms)
2015-02-20 16:43:00 +1100: slow request (1083 ms)
2015-02-20 16:44:15 +1100: 60 successful reqs
2015-02-20 16:45:50 +1100: 60 successful reqs
2015-02-20 16:46:31 +1100: slow request (1360 ms)
2015-02-20 16:46:33 +1100: slow request (1666 ms)
2015-02-20 16:47:08 +1100: slow request (1049 ms)
2015-02-20 16:47:28 +1100: 60 successful reqs
2015-02-20 16:49:03 +1100: 60 successful reqs
2015-02-20 16:50:37 +1100: 60 successful reqs
2015-02-20 16:52:12 +1100: 60 successful reqs
2015-02-20 16:53:47 +1100: 60 successful reqs
2015-02-20 16:55:23 +1100: 60 successful reqs
2015-02-20 16:56:57 +1100: 60 successful reqs
2015-02-20 16:58:32 +1100: 60 successful reqs
2015-02-20 17:00:07 +1100: 60 successful reqs
2015-02-20 17:01:27 +1100: slow request (1208 ms)
2015-02-20 17:01:44 +1100: 60 successful reqs
2015-02-20 17:02:33 +1100: slow request (1147 ms)
2015-02-20 17:03:20 +1100: 60 successful reqs
2015-02-20 17:04:55 +1100: 60 successful reqs
2015-02-20 17:06:31 +1100: 60 successful reqs
2015-02-20 17:08:05 +1100: 60 successful reqs
2015-02-20 17:09:41 +1100: 60 successful reqs
2015-02-20 17:11:17 +1100: 60 successful reqs
2015-02-20 17:12:52 +1100: 60 successful reqs
2015-02-20 17:14:19 +1100: slow request (1098 ms)
2015-02-20 17:14:30 +1100: 60 successful reqs
2015-02-20 17:16:06 +1100: 60 successful reqs
2015-02-20 17:17:41 +1100: 60 successful reqs
2015-02-20 17:19:18 +1100: 60 successful reqs
2015-02-20 17:20:54 +1100: 60 successful reqs
2015-02-20 17:22:31 +1100: 60 successful reqs
2015-02-20 17:24:07 +1100: 60 successful reqs
2015-02-20 17:25:42 +1100: 60 successful reqs
2015-02-20 17:27:18 +1100: 60 successful reqs
2015-02-20 17:28:53 +1100: 60 successful reqs
2015-02-20 17:30:29 +1100: 60 successful reqs
2015-02-20 17:32:06 +1100: 60 successful reqs
2015-02-20 17:33:41 +1100: 60 successful reqs
2015-02-20 17:35:15 +1100: slow request (1209 ms)
2015-02-20 17:35:20 +1100: 60 successful reqs
2015-02-20 17:36:55 +1100: 60 successful reqs
2015-02-20 17:38:22 +1100: slow request (1814 ms)
2015-02-20 17:38:26 +1100: slow request (2680 ms)
2015-02-20 17:38:30 +1100: slow request (2937 ms)
2015-02-20 17:38:32 +1100: slow request (1250 ms)
2015-02-20 17:38:37 +1100: 60 successful reqs
2015-02-20 17:40:12 +1100: 60 successful reqs
2015-02-20 17:41:48 +1100: 60 successful reqs
2015-02-20 17:43:24 +1100: 60 successful reqs
2015-02-20 17:44:59 +1100: 60 successful reqs
2015-02-20 17:46:34 +1100: 60 successful reqs
2015-02-20 17:48:09 +1100: 60 successful reqs
2015-02-20 17:49:44 +1100: 60 successful reqs
2015-02-20 17:51:21 +1100: 60 successful reqs
2015-02-20 17:52:56 +1100: 60 successful reqs
2015-02-20 17:54:19 +1100: slow request (5521 ms)
2015-02-20 17:54:36 +1100: 60 successful reqs
2015-02-20 17:56:10 +1100: 60 successful reqs
2015-02-20 17:57:38 +1100: slow request (1219 ms)
2015-02-20 17:57:45 +1100: 60 successful reqs
2015-02-20 17:59:20 +1100: 60 successful reqs
2015-02-20 18:00:55 +1100: 60 successful reqs
2015-02-20 18:02:30 +1100: 60 successful reqs
2015-02-20 18:04:05 +1100: 60 successful reqs
2015-02-20 18:05:40 +1100: 60 successful reqs
2015-02-20 18:07:15 +1100: 60 successful reqs
2015-02-20 18:08:50 +1100: 60 successful reqs
2015-02-20 18:10:26 +1100: 60 successful reqs
2015-02-20 18:12:00 +1100: 60 successful reqs
2015-02-20 18:13:36 +1100: 60 successful reqs
2015-02-20 18:15:12 +1100: 60 successful reqs
2015-02-20 18:16:50 +1100: 60 successful reqs
2015-02-20 18:18:28 +1100: 60 successful reqs
2015-02-20 18:20:04 +1100: 60 successful reqs
2015-02-20 18:20:25 +1100: slow request (1119 ms)
2015-02-20 18:21:52 +1100: 60 successful reqs
2015-02-20 18:22:56 +1100: slow request (1142 ms)
2015-02-20 18:23:30 +1100: 60 successful reqs
2015-02-20 18:23:53 +1100: slow request (5595 ms)
2015-02-20 18:25:13 +1100: 60 successful reqs
2015-02-20 18:26:51 +1100: 60 successful reqs
2015-02-20 18:27:16 +1100: slow request (1229 ms)
2015-02-20 18:28:30 +1100: 60 successful reqs
2015-02-20 18:30:08 +1100: 60 successful reqs
2015-02-20 18:30:47 +1100: slow request (1290 ms)
2015-02-20 18:30:50 +1100: slow request (2046 ms)
2015-02-20 18:30:52 +1100: slow request (1360 ms)

Besides a few of these (which we should sort out)

 duration: 1348.898 ms  statement: UPDATE posts SET percent_rank = x.percent_rank
	              FROM (SELECT id, percent_rank()
	                    OVER (PARTITION BY topic_id ORDER BY SCORE DESC) as percent_rank
	                    FROM posts) AS x
	                   WHERE (x.id = posts.id AND
	               (posts.percent_rank IS NULL OR x.percent_rank <> posts.percent_rank)) AND (posts.topic_id IN
	                    (SELECT id FROM topics WHERE bumped_at > '2015-02-19 07:23:45.067340'))

and this from the user page

05:47:10 UTC LOG:  duration: 1147.734 ms  execute <unnamed>: 
	    SELECT action_type, COUNT(*) count
	    FROM user_actions a
	    JOIN topics t ON t.id = a.target_topic_id
	    LEFT JOIN posts p on p.id = a.target_post_id
	    JOIN posts p2 on p2.topic_id = a.target_topic_id and p2.post_number = 1
	    LEFT JOIN categories c ON c.id = t.category_id
	    WHERE (a.user_id = 603) AND (t.deleted_at is null) AND (p.deleted_at is null and p2.deleted_at is null) AND (NOT COALESCE(p.hidden, false) OR p.user_id = 914) AND (a.action_type not in (3)) AND (t.visible) AND (t.archetype != 'private_message') AND (( c.read_restricted IS NULL OR
	                         NOT c.read_restricted OR
	                        (c.read_restricted and c.id in (23,26,28,28,30,30)) ))
	    GROUP BY action_type

the majority of the queries that are slow are all the badge queries

[Onyx]

@sam said:

materialized views in postgres kind of suck cause you need to keep issuing refreshes to them, but I guess it could be done, if a plugin was added that does that.

Trigger? I'm not sure how expensive refreshing will be given the data though.

@Weng said:

Build temp tables and join on those

Remember, queries were made by @PJH and added from Discourse's GUI. He doesn't have the permissions to do that.

@Weng said:

Or add a table and manually update it with topic ids

Again, no permissions...

[sam]

@Onyx said:

Trigger?

refreshing this kind of stuff can be expensive but my bigger concern is that it introduces magic schema that is only here and coupled to the posts table.

Instead a simple and efficient solution here is to create a sock puppet and have it "like" these topics you want ignored, then you have a super cheap thing to join to. (since it would only like post #1 on these ignored topics) - alternatively have that sock mute the topics and then you can use topic users for that user.

[riking]

Or hijack that user's bio or location field to store a comma-separated list of topic IDs.

[Onyx]

@sam said:

Instead a simple and efficient solution here is to create a sock puppet and have it "like" these topics you want ignored

You can "Like" a topic? Or do you mean the OP?

[sam]

yeah. like the OP, this sock would only be liking the OP anyway so post_actions for it would be tiny.

[Onyx]

Sounds good, to me at least.

@PJH?

[PJH]

@sam said:

OR if you can just run it daily instead.

This.

[PJH]

@jaming said:

Killing that LIKE should be a marked improvement, I'd think

That part of the query is sub-second.

@Monarch said:

INSERT INTO user_badges(badge_id, user_id, granted_at, granted_by_id, post_id)

That isn't part of the query to which I have access.

@sam said:

Instead a simple and efficient solution here is to create a sock puppet and have it "like" these topics you want ignored,

Ok - that takes care of one of the UUIDs. What about the others?

[accalia]

@PJH said:

What about the others?

Moar sock puppets?

[PJH]

@accalia said:

Moar sock puppets?

That's going to scale well...

[accalia]

Fair enough....

[tarunik]

@Weng said:

Too much nesting. Build temp tables and join on those, and trade where-in statements for joins (yeah the query optimizer might do the right thing anyway, but it's an asshole and will change its mind to your detriment in future)

I'll defer on this one; I don't deal with pg much and really am not good with query optimizers

@Weng said:

Paranoid locking of tables that are read will cause excess blocking. If you're not going to fuck with the table and it isn't timing critical, hint away the locks or run at snapshot isolation.

But this is inapplicable. PostgreSQL uses Multi-Version Concurrency Control (MVCC) -- so there's really no way to kill yourself with lock paranoia just by reading tables.

[Monarch]

This post is deleted!

[abarker]

@Polygeekery said:

Or, adding functionality to Discourse, which would be fraught with peril...

The most effective method would require a DB change, and I don't think that @PJH wants to mess with that, or that the discodevs are going to accept such a request.

[PleegWat]

Or the team would need to add a topic exclusion mechanism to the base product.

Something else that might help with badge queries is something so an entire series like 3^n could be done in one query, given that the badge-specific requirement for those is the very last part applied. But I think all of those are nightly so it's not as much of a restriction.

[aliceif]

@PleegWat said:

topic exclusion mechanism

tag-based, maybe?

[abarker]

@aliceif said:

tag-based, maybe?

In that case, tags would need to be restricted somehow so that TL3- couldn't remove the exclusion.

[boomzilla]

@PleegWat said:

Or the team would need to add a topic exclusion mechanism to the base product.

This is really probably the least problematic part of the query, performance-wise. It looks like it shouldn't, but there's some kind of indexing happening that makes it pretty trivial.

[PleegWat]

Yeah, still everyone's blaming it.

The 'topics with less than 10 posts' part, where it exists, seems more likely.

[sam]

@PJH still seeing those queries, mind changing them to daily till we work out a more efficient way to write them.

[PJH]

... I thought I had changed them. All the bronze 2^n I changed from per edit to per day...

Which did I miss? (I'll check in an hour when I get to a PC)

[PleegWat]

@sam said:

GROUP BY user_id HAVING count(*) >= POW(2, 0)

2^0

EDIT: @PJH

[PJH]

What about it?

[PleegWat]

That's the query @sam quoted as running on each post. Unless you changed that too daily after he listed the queries running on post, something is up...

[PJH]

Yup, all the bronze 2^n badges were running per post, now (should be) running daily.

[Groaner]

@HardwareGeek said:

I may be wrong, but I'm under the impression @PJH doesn't have the access to do that.

@sam said:

Temp tables would need a plugin to populate them.

@Onyx said:

Remember, queries were made by @PJH and added from Discourse's GUI. He doesn't have the permissions to do that.

What? You need permissions to create temp tables in postgres?

[Yamikuronue]

Generally to create anything you need access to the database. PJH only has access to about half a query in a text field IIRC

[Groaner]

@Yamikuronue said:

Generally to create anything you need access to the database. PJH only has access to about half a query in a text field IIRC

By half a query, do you mean one can only manipulate joins and WHERE clauses or something?

[PleegWat]

It's actually a subquery so he can't make it an insert or anything.

[Yamikuronue]

That's my understanding, yeah. I'm sure PJH will come by and correct it when he's done being hungover ;)

[Groaner]

Ok, that makes sense.

[sam]

except for 2^13, sorted that out, not many queries taking longer than 1 second now

[sam]

Now that a lot of the heavy lifting SQL is slowed down my monitoring is showing stuff to be much less cootie-ie

Very few queries are meeting the 1 second threshold for logging.

[sam]

Logs from 8pm last night till 9am this morning:

2015-02-22 20:31:05 +1100: 60 successful reqs
2015-02-22 20:32:41 +1100: 60 successful reqs
2015-02-22 20:34:18 +1100: 60 successful reqs
2015-02-22 20:35:55 +1100: 60 successful reqs
2015-02-22 20:37:30 +1100: 60 successful reqs
2015-02-22 20:39:06 +1100: 60 successful reqs
2015-02-22 20:40:42 +1100: 60 successful reqs
2015-02-22 20:42:18 +1100: 60 successful reqs
2015-02-22 20:43:53 +1100: 60 successful reqs
2015-02-22 20:45:29 +1100: 60 successful reqs
2015-02-22 20:47:05 +1100: 60 successful reqs
2015-02-22 20:48:40 +1100: 60 successful reqs
2015-02-22 20:50:15 +1100: 60 successful reqs
2015-02-22 20:51:51 +1100: 60 successful reqs
2015-02-22 20:53:26 +1100: 60 successful reqs
2015-02-22 20:55:02 +1100: 60 successful reqs
2015-02-22 20:55:20 +1100: slow request (1228 ms)
2015-02-22 20:56:38 +1100: 60 successful reqs
2015-02-22 20:58:13 +1100: 60 successful reqs
2015-02-22 20:59:48 +1100: 60 successful reqs
2015-02-22 21:01:23 +1100: 60 successful reqs
2015-02-22 21:02:59 +1100: 60 successful reqs
2015-02-22 21:03:24 +1100: slow request (1099 ms)
2015-02-22 21:04:35 +1100: 60 successful reqs
2015-02-22 21:06:10 +1100: 60 successful reqs
2015-02-22 21:07:46 +1100: 60 successful reqs
2015-02-22 21:09:21 +1100: 60 successful reqs
2015-02-22 21:10:57 +1100: 60 successful reqs
2015-02-22 21:12:32 +1100: 60 successful reqs
2015-02-22 21:13:07 +1100: slow request (2277 ms)
2015-02-22 21:14:09 +1100: 60 successful reqs
2015-02-22 21:15:44 +1100: 60 successful reqs
2015-02-22 21:15:55 +1100: slow request (1612 ms)
2015-02-22 21:16:22 +1100: slow request (1095 ms)
2015-02-22 21:16:33 +1100: slow request (1118 ms)
2015-02-22 21:16:35 +1100: slow request (1257 ms)
2015-02-22 21:17:22 +1100: 60 successful reqs
2015-02-22 21:18:57 +1100: 60 successful reqs
2015-02-22 21:20:33 +1100: 60 successful reqs
2015-02-22 21:21:11 +1100: slow request (1007 ms)
2015-02-22 21:22:09 +1100: 60 successful reqs
2015-02-22 21:23:44 +1100: 60 successful reqs
2015-02-22 21:25:20 +1100: 60 successful reqs
2015-02-22 21:26:51 +1100: slow request (1056 ms)
2015-02-22 21:26:57 +1100: 60 successful reqs
2015-02-22 21:28:33 +1100: 60 successful reqs
2015-02-22 21:30:09 +1100: 60 successful reqs
2015-02-22 21:31:44 +1100: 60 successful reqs
2015-02-22 21:33:20 +1100: 60 successful reqs
2015-02-22 21:34:55 +1100: 60 successful reqs
2015-02-22 21:36:30 +1100: 60 successful reqs
2015-02-22 21:38:05 +1100: 60 successful reqs
2015-02-22 21:39:40 +1100: 60 successful reqs
2015-02-22 21:41:16 +1100: 60 successful reqs
2015-02-22 21:42:52 +1100: 60 successful reqs
2015-02-22 21:44:27 +1100: 60 successful reqs
2015-02-22 21:46:02 +1100: 60 successful reqs
2015-02-22 21:47:37 +1100: 60 successful reqs
2015-02-22 21:49:13 +1100: 60 successful reqs
2015-02-22 21:50:48 +1100: 60 successful reqs
2015-02-22 21:52:23 +1100: 60 successful reqs
2015-02-22 21:53:59 +1100: 60 successful reqs
2015-02-22 21:55:34 +1100: 60 successful reqs
2015-02-22 21:57:09 +1100: 60 successful reqs
2015-02-22 21:58:44 +1100: 60 successful reqs
2015-02-22 22:00:20 +1100: 60 successful reqs
2015-02-22 22:01:56 +1100: 60 successful reqs
2015-02-22 22:03:31 +1100: 60 successful reqs
2015-02-22 22:05:07 +1100: 60 successful reqs
2015-02-22 22:06:42 +1100: 60 successful reqs
2015-02-22 22:08:20 +1100: slow request (2286 ms)
2015-02-22 22:08:20 +1100: 60 successful reqs
2015-02-22 22:09:56 +1100: 60 successful reqs
2015-02-22 22:11:31 +1100: 60 successful reqs
2015-02-22 22:13:07 +1100: 60 successful reqs
2015-02-22 22:14:43 +1100: 60 successful reqs
2015-02-22 22:16:18 +1100: 60 successful reqs
2015-02-22 22:17:54 +1100: 60 successful reqs
2015-02-22 22:19:30 +1100: 60 successful reqs
2015-02-22 22:21:06 +1100: 60 successful reqs
2015-02-22 22:22:41 +1100: 60 successful reqs
2015-02-22 22:24:17 +1100: 60 successful reqs
2015-02-22 22:25:52 +1100: 60 successful reqs
2015-02-22 22:27:28 +1100: 60 successful reqs
2015-02-22 22:29:04 +1100: 60 successful reqs
2015-02-22 22:30:40 +1100: 60 successful reqs
2015-02-22 22:32:16 +1100: 60 successful reqs
2015-02-22 22:33:51 +1100: 60 successful reqs
2015-02-22 22:35:27 +1100: 60 successful reqs
2015-02-22 22:37:01 +1100: slow request (5509 ms)
2015-02-22 22:37:07 +1100: 60 successful reqs
2015-02-22 22:38:43 +1100: 60 successful reqs
2015-02-22 22:40:18 +1100: 60 successful reqs
2015-02-22 22:41:54 +1100: 60 successful reqs
2015-02-22 22:43:29 +1100: 60 successful reqs
2015-02-22 22:45:05 +1100: 60 successful reqs
2015-02-22 22:46:40 +1100: 60 successful reqs
2015-02-22 22:48:16 +1100: 60 successful reqs
2015-02-22 22:49:51 +1100: 60 successful reqs
2015-02-22 22:51:27 +1100: 60 successful reqs
2015-02-22 22:53:03 +1100: 60 successful reqs
2015-02-22 22:54:39 +1100: 60 successful reqs
2015-02-22 22:56:14 +1100: 60 successful reqs
2015-02-22 22:57:51 +1100: 60 successful reqs
2015-02-22 22:59:26 +1100: 60 successful reqs
2015-02-22 23:00:37 +1100: slow request (1035 ms)
2015-02-22 23:01:02 +1100: 60 successful reqs
2015-02-22 23:02:38 +1100: 60 successful reqs
2015-02-22 23:04:15 +1100: 60 successful reqs
2015-02-22 23:05:50 +1100: 60 successful reqs
2015-02-22 23:07:25 +1100: 60 successful reqs
2015-02-22 23:09:01 +1100: 60 successful reqs
2015-02-22 23:10:36 +1100: 60 successful reqs
2015-02-22 23:11:25 +1100: slow request (2186 ms)
2015-02-22 23:11:27 +1100: slow request (1133 ms)
2015-02-22 23:11:29 +1100: slow request (1031 ms)
2015-02-22 23:11:42 +1100: slow request (1381 ms)
2015-02-22 23:11:49 +1100: slow request (1132 ms)
2015-02-22 23:12:04 +1100: slow request (1141 ms)
2015-02-22 23:12:18 +1100: 60 successful reqs
2015-02-22 23:12:24 +1100: slow request (1377 ms)
2015-02-22 23:13:02 +1100: slow request (1081 ms)
2015-02-22 23:13:04 +1100: slow request (1114 ms)
2015-02-22 23:13:56 +1100: 60 successful reqs
2015-02-22 23:14:37 +1100: slow request (1168 ms)
2015-02-22 23:14:39 +1100: slow request (1112 ms)
2015-02-22 23:15:32 +1100: 60 successful reqs
2015-02-22 23:17:08 +1100: 60 successful reqs
2015-02-22 23:18:43 +1100: 60 successful reqs
2015-02-22 23:20:19 +1100: 60 successful reqs
2015-02-22 23:21:54 +1100: 60 successful reqs
2015-02-22 23:23:29 +1100: 60 successful reqs
2015-02-22 23:25:04 +1100: 60 successful reqs
2015-02-22 23:26:41 +1100: 60 successful reqs
2015-02-22 23:28:17 +1100: 60 successful reqs
2015-02-22 23:29:52 +1100: 60 successful reqs
2015-02-22 23:31:28 +1100: 60 successful reqs
2015-02-22 23:33:03 +1100: 60 successful reqs
2015-02-22 23:34:39 +1100: 60 successful reqs
2015-02-22 23:36:14 +1100: 60 successful reqs
2015-02-22 23:37:49 +1100: 60 successful reqs
2015-02-22 23:39:25 +1100: 60 successful reqs
2015-02-22 23:41:01 +1100: 60 successful reqs
2015-02-22 23:42:39 +1100: 60 successful reqs
2015-02-22 23:44:15 +1100: 60 successful reqs
2015-02-22 23:45:51 +1100: 60 successful reqs
2015-02-22 23:47:27 +1100: 60 successful reqs
2015-02-22 23:49:03 +1100: 60 successful reqs
2015-02-22 23:50:39 +1100: 60 successful reqs
2015-02-22 23:52:15 +1100: 60 successful reqs
2015-02-22 23:53:13 +1100: slow request (1597 ms)
2015-02-22 23:53:15 +1100: slow request (1153 ms)
2015-02-22 23:53:52 +1100: 60 successful reqs
2015-02-22 23:55:28 +1100: 60 successful reqs
2015-02-22 23:55:33 +1100: slow request (1213 ms)
2015-02-22 23:56:54 +1100: slow request (1233 ms)
2015-02-22 23:57:05 +1100: 60 successful reqs
2015-02-22 23:58:41 +1100: 60 successful reqs
2015-02-23 00:00:19 +1100: 60 successful reqs
2015-02-23 00:01:55 +1100: 60 successful reqs
2015-02-23 00:03:30 +1100: 60 successful reqs
2015-02-23 00:05:07 +1100: 60 successful reqs
2015-02-23 00:06:43 +1100: slow request (1065 ms)
2015-02-23 00:06:43 +1100: 60 successful reqs
2015-02-23 00:06:49 +1100: slow request (1025 ms)
2015-02-23 00:06:51 +1100: slow request (1270 ms)
2015-02-23 00:06:58 +1100: slow request (1280 ms)
2015-02-23 00:07:00 +1100: slow request (1110 ms)
2015-02-23 00:08:22 +1100: 60 successful reqs
2015-02-23 00:09:58 +1100: 60 successful reqs
2015-02-23 00:10:34 +1100: slow request (1644 ms)
2015-02-23 00:11:36 +1100: 60 successful reqs
2015-02-23 00:13:12 +1100: 60 successful reqs
2015-02-23 00:14:47 +1100: 60 successful reqs
2015-02-23 00:16:23 +1100: 60 successful reqs
2015-02-23 00:17:59 +1100: 60 successful reqs
2015-02-23 00:19:34 +1100: 60 successful reqs
2015-02-23 00:21:10 +1100: 60 successful reqs
2015-02-23 00:22:46 +1100: 60 successful reqs
2015-02-23 00:24:22 +1100: 60 successful reqs
2015-02-23 00:25:58 +1100: 60 successful reqs
2015-02-23 00:27:34 +1100: 60 successful reqs
2015-02-23 00:28:48 +1100: slow request (1063 ms)
2015-02-23 00:29:11 +1100: 60 successful reqs
2015-02-23 00:30:48 +1100: 60 successful reqs
2015-02-23 00:32:24 +1100: 60 successful reqs
2015-02-23 00:34:00 +1100: 60 successful reqs
2015-02-23 00:35:36 +1100: 60 successful reqs
2015-02-23 00:35:58 +1100: slow request (1030 ms)
2015-02-23 00:37:12 +1100: 60 successful reqs
2015-02-23 00:38:49 +1100: 60 successful reqs
2015-02-23 00:39:55 +1100: slow request (1067 ms)
2015-02-23 00:40:01 +1100: slow request (1197 ms)
2015-02-23 00:40:26 +1100: 60 successful reqs
2015-02-23 00:42:01 +1100: 60 successful reqs
2015-02-23 00:43:37 +1100: 60 successful reqs
2015-02-23 00:45:12 +1100: 60 successful reqs
2015-02-23 00:46:47 +1100: 60 successful reqs
2015-02-23 00:48:22 +1100: 60 successful reqs
2015-02-23 00:49:57 +1100: 60 successful reqs
2015-02-23 00:51:33 +1100: 60 successful reqs
2015-02-23 00:53:09 +1100: 60 successful reqs
2015-02-23 00:53:50 +1100: slow request (2424 ms)
2015-02-23 00:54:46 +1100: 60 successful reqs
2015-02-23 00:55:02 +1100: slow request (1049 ms)
2015-02-23 00:55:05 +1100: slow request (1476 ms)
2015-02-23 00:56:23 +1100: 60 successful reqs
2015-02-23 00:57:59 +1100: 60 successful reqs
2015-02-23 00:59:34 +1100: 60 successful reqs
2015-02-23 01:01:10 +1100: 60 successful reqs
2015-02-23 01:02:45 +1100: 60 successful reqs
2015-02-23 01:04:21 +1100: 60 successful reqs
2015-02-23 01:05:56 +1100: 60 successful reqs
2015-02-23 01:07:31 +1100: 60 successful reqs
2015-02-23 01:09:06 +1100: 60 successful reqs
2015-02-23 01:10:12 +1100: slow request (1457 ms)
2015-02-23 01:10:43 +1100: 60 successful reqs
2015-02-23 01:12:18 +1100: 60 successful reqs
2015-02-23 01:13:54 +1100: 60 successful reqs
2015-02-23 01:14:38 +1100: slow request (1267 ms)
2015-02-23 01:15:29 +1100: 60 successful reqs
2015-02-23 01:16:14 +1100: slow request (1044 ms)
2015-02-23 01:17:05 +1100: 60 successful reqs
2015-02-23 01:18:41 +1100: 60 successful reqs
2015-02-23 01:20:16 +1100: 60 successful reqs
2015-02-23 01:21:51 +1100: 60 successful reqs
2015-02-23 01:23:27 +1100: 60 successful reqs
2015-02-23 01:25:03 +1100: 60 successful reqs
2015-02-23 01:26:38 +1100: 60 successful reqs
2015-02-23 01:28:14 +1100: 60 successful reqs
2015-02-23 01:29:13 +1100: slow request (1026 ms)
2015-02-23 01:29:28 +1100: slow request (1460 ms)
2015-02-23 01:29:52 +1100: 60 successful reqs
2015-02-23 01:31:28 +1100: 60 successful reqs
2015-02-23 01:33:03 +1100: 60 successful reqs
2015-02-23 01:34:38 +1100: 60 successful reqs
2015-02-23 01:36:14 +1100: 60 successful reqs
2015-02-23 01:37:50 +1100: 60 successful reqs
2015-02-23 01:39:26 +1100: 60 successful reqs
2015-02-23 01:41:01 +1100: 60 successful reqs
2015-02-23 01:42:36 +1100: 60 successful reqs
2015-02-23 01:44:11 +1100: 60 successful reqs
2015-02-23 01:45:46 +1100: 60 successful reqs
2015-02-23 01:47:21 +1100: 60 successful reqs
2015-02-23 01:48:35 +1100: slow request (2364 ms)
2015-02-23 01:48:58 +1100: 60 successful reqs
2015-02-23 01:50:33 +1100: 60 successful reqs
2015-02-23 01:52:09 +1100: 60 successful reqs
2015-02-23 01:53:44 +1100: 60 successful reqs
2015-02-23 01:55:20 +1100: 60 successful reqs
2015-02-23 01:56:55 +1100: 60 successful reqs
2015-02-23 01:58:30 +1100: 60 successful reqs
2015-02-23 02:00:06 +1100: 60 successful reqs
2015-02-23 02:01:42 +1100: 60 successful reqs
2015-02-23 02:03:17 +1100: 60 successful reqs
2015-02-23 02:04:52 +1100: 60 successful reqs
2015-02-23 02:06:27 +1100: 60 successful reqs
2015-02-23 02:08:02 +1100: 60 successful reqs
2015-02-23 02:09:37 +1100: 60 successful reqs
2015-02-23 02:11:13 +1100: 60 successful reqs
2015-02-23 02:12:49 +1100: 60 successful reqs
2015-02-23 02:13:03 +1100: slow request (2181 ms)
2015-02-23 02:13:06 +1100: slow request (2252 ms)
2015-02-23 02:14:27 +1100: 60 successful reqs
2015-02-23 02:16:02 +1100: 60 successful reqs
2015-02-23 02:17:38 +1100: 60 successful reqs
2015-02-23 02:19:13 +1100: 60 successful reqs
2015-02-23 02:20:49 +1100: 60 successful reqs
2015-02-23 02:22:25 +1100: 60 successful reqs
2015-02-23 02:24:00 +1100: 60 successful reqs
2015-02-23 02:25:34 +1100: slow request (1015 ms)
2015-02-23 02:25:36 +1100: 60 successful reqs
2015-02-23 02:27:12 +1100: 60 successful reqs
2015-02-23 02:28:47 +1100: 60 successful reqs
2015-02-23 02:30:23 +1100: 60 successful reqs
2015-02-23 02:31:58 +1100: 60 successful reqs
2015-02-23 02:33:34 +1100: 60 successful reqs
2015-02-23 02:35:10 +1100: 60 successful reqs
2015-02-23 02:35:20 +1100: slow request (1500 ms)
2015-02-23 02:35:22 +1100: slow request (1058 ms)
2015-02-23 02:35:25 +1100: slow request (2112 ms)
2015-02-23 02:35:28 +1100: slow request (1306 ms)
2015-02-23 02:35:31 +1100: slow request (1187 ms)
2015-02-23 02:35:49 +1100: slow request (1288 ms)
2015-02-23 02:35:52 +1100: slow request (1119 ms)
2015-02-23 02:36:51 +1100: 60 successful reqs
2015-02-23 02:37:40 +1100: slow request (1445 ms)
2015-02-23 02:37:42 +1100: slow request (1002 ms)
2015-02-23 02:37:54 +1100: slow request (1172 ms)
2015-02-23 02:38:14 +1100: slow request (1309 ms)
2015-02-23 02:38:30 +1100: 60 successful reqs
2015-02-23 02:40:05 +1100: 60 successful reqs
2015-02-23 02:41:40 +1100: 60 successful reqs
2015-02-23 02:43:15 +1100: 60 successful reqs
2015-02-23 02:44:52 +1100: 60 successful reqs
2015-02-23 02:46:26 +1100: 60 successful reqs
2015-02-23 02:48:02 +1100: 60 successful reqs
2015-02-23 02:49:19 +1100: slow request (1135 ms)
2015-02-23 02:49:38 +1100: 60 successful reqs
2015-02-23 02:51:13 +1100: 60 successful reqs
2015-02-23 02:52:49 +1100: 60 successful reqs
2015-02-23 02:54:25 +1100: 60 successful reqs
2015-02-23 02:56:00 +1100: 60 successful reqs
2015-02-23 02:57:36 +1100: 60 successful reqs
2015-02-23 02:59:11 +1100: 60 successful reqs
2015-02-23 03:00:47 +1100: 60 successful reqs
2015-02-23 03:02:22 +1100: 60 successful reqs
2015-02-23 03:03:57 +1100: 60 successful reqs
2015-02-23 03:04:46 +1100: slow request (1308 ms)
2015-02-23 03:04:49 +1100: slow request (1655 ms)
2015-02-23 03:05:35 +1100: 60 successful reqs
2015-02-23 03:07:10 +1100: 60 successful reqs
2015-02-23 03:08:46 +1100: 60 successful reqs
2015-02-23 03:10:22 +1100: 60 successful reqs
2015-02-23 03:11:58 +1100: 60 successful reqs
2015-02-23 03:13:33 +1100: 60 successful reqs
2015-02-23 03:15:08 +1100: 60 successful reqs
2015-02-23 03:16:44 +1100: 60 successful reqs
2015-02-23 03:18:19 +1100: 60 successful reqs
2015-02-23 03:19:55 +1100: 60 successful reqs
2015-02-23 03:21:31 +1100: 60 successful reqs
2015-02-23 03:23:07 +1100: 60 successful reqs
2015-02-23 03:24:02 +1100: slow request (1157 ms)
2015-02-23 03:24:43 +1100: 60 successful reqs
2015-02-23 03:26:18 +1100: 60 successful reqs
2015-02-23 03:27:54 +1100: 60 successful reqs
2015-02-23 03:29:30 +1100: 60 successful reqs
2015-02-23 03:31:06 +1100: 60 successful reqs
2015-02-23 03:32:41 +1100: 60 successful reqs
2015-02-23 03:34:17 +1100: 60 successful reqs
2015-02-23 03:35:52 +1100: 60 successful reqs
2015-02-23 03:37:27 +1100: slow request (5522 ms)
2015-02-23 03:37:33 +1100: 60 successful reqs
2015-02-23 03:39:11 +1100: 60 successful reqs
2015-02-23 03:40:46 +1100: 60 successful reqs
2015-02-23 03:42:21 +1100: 60 successful reqs
2015-02-23 03:43:57 +1100: 60 successful reqs
2015-02-23 03:45:32 +1100: 60 successful reqs
2015-02-23 03:47:08 +1100: 60 successful reqs
2015-02-23 03:48:43 +1100: 60 successful reqs
2015-02-23 03:50:18 +1100: 60 successful reqs
2015-02-23 03:51:54 +1100: 60 successful reqs
2015-02-23 03:53:29 +1100: 60 successful reqs
2015-02-23 03:55:05 +1100: 60 successful reqs
2015-02-23 03:56:40 +1100: 60 successful reqs
2015-02-23 03:58:15 +1100: 60 successful reqs
2015-02-23 03:59:51 +1100: 60 successful reqs
2015-02-23 04:01:27 +1100: 60 successful reqs
2015-02-23 04:01:43 +1100: slow request (1610 ms)
2015-02-23 04:02:04 +1100: slow request (1606 ms)
2015-02-23 04:02:06 +1100: slow request (1158 ms)
2015-02-23 04:02:26 +1100: slow request (1381 ms)
2015-02-23 04:03:07 +1100: 60 successful reqs
2015-02-23 04:03:14 +1100: slow request (1436 ms)
2015-02-23 04:04:43 +1100: 60 successful reqs
2015-02-23 04:06:05 +1100: slow request (1312 ms)
2015-02-23 04:06:09 +1100: slow request (1114 ms)
2015-02-23 04:06:21 +1100: 60 successful reqs
2015-02-23 04:07:57 +1100: 60 successful reqs
2015-02-23 04:09:33 +1100: 60 successful reqs
2015-02-23 04:11:08 +1100: 60 successful reqs
2015-02-23 04:12:43 +1100: 60 successful reqs
2015-02-23 04:14:19 +1100: 60 successful reqs
2015-02-23 04:15:54 +1100: 60 successful reqs
2015-02-23 04:17:29 +1100: 60 successful reqs
2015-02-23 04:19:06 +1100: 60 successful reqs
2015-02-23 04:20:42 +1100: 60 successful reqs
2015-02-23 04:22:17 +1100: 60 successful reqs
2015-02-23 04:23:53 +1100: 60 successful reqs
2015-02-23 04:25:28 +1100: 60 successful reqs
2015-02-23 04:27:03 +1100: 60 successful reqs
2015-02-23 04:28:39 +1100: 60 successful reqs
2015-02-23 04:30:15 +1100: 60 successful reqs
2015-02-23 04:31:51 +1100: 60 successful reqs
2015-02-23 04:33:27 +1100: 60 successful reqs
2015-02-23 04:34:20 +1100: slow request (2434 ms)
2015-02-23 04:35:04 +1100: 60 successful reqs
2015-02-23 04:36:39 +1100: 60 successful reqs
2015-02-23 04:38:14 +1100: 60 successful reqs
2015-02-23 04:39:05 +1100: slow request (1189 ms)
2015-02-23 04:39:51 +1100: 60 successful reqs
2015-02-23 04:41:28 +1100: slow request (1420 ms)
2015-02-23 04:41:28 +1100: 60 successful reqs
2015-02-23 04:43:05 +1100: 60 successful reqs
2015-02-23 04:44:01 +1100: slow request (1236 ms)
2015-02-23 04:44:05 +1100: slow request (3120 ms)
2015-02-23 04:44:44 +1100: 60 successful reqs
2015-02-23 04:46:07 +1100: slow request (1099 ms)
2015-02-23 04:46:21 +1100: 60 successful reqs
2015-02-23 04:46:38 +1100: slow request (1376 ms)
2015-02-23 04:47:59 +1100: 60 successful reqs
2015-02-23 04:48:47 +1100: slow request (1057 ms)
2015-02-23 04:49:36 +1100: 60 successful reqs
2015-02-23 04:50:35 +1100: slow request (1104 ms)
2015-02-23 04:51:13 +1100: 60 successful reqs
2015-02-23 04:51:17 +1100: slow request (2078 ms)
2015-02-23 04:51:22 +1100: slow request (3310 ms)
2015-02-23 04:51:24 +1100: slow request (1575 ms)
2015-02-23 04:51:27 +1100: slow request (1467 ms)
2015-02-23 04:51:56 +1100: slow request (1399 ms)
2015-02-23 04:52:55 +1100: 60 successful reqs
2015-02-23 04:54:31 +1100: 60 successful reqs
2015-02-23 04:56:06 +1100: 60 successful reqs
2015-02-23 04:57:42 +1100: 60 successful reqs
2015-02-23 04:59:18 +1100: 60 successful reqs
2015-02-23 05:00:42 +1100: slow request (1001 ms)
2015-02-23 05:00:45 +1100: slow request (1963 ms)
2015-02-23 05:00:47 +1100: slow request (1311 ms)
2015-02-23 05:00:57 +1100: 60 successful reqs
2015-02-23 05:02:32 +1100: 60 successful reqs
2015-02-23 05:04:09 +1100: 60 successful reqs
2015-02-23 05:05:44 +1100: 60 successful reqs
2015-02-23 05:07:20 +1100: 60 successful reqs
2015-02-23 05:08:52 +1100: slow request (1701 ms)
2015-02-23 05:08:55 +1100: slow request (1792 ms)
2015-02-23 05:08:57 +1100: slow request (1364 ms)
2015-02-23 05:08:59 +1100: 60 successful reqs
2015-02-23 05:09:20 +1100: slow request (1179 ms)
2015-02-23 05:10:35 +1100: 60 successful reqs
2015-02-23 05:11:27 +1100: slow request (1078 ms)
2015-02-23 05:11:33 +1100: slow request (1194 ms)
2015-02-23 05:11:58 +1100: slow request (2296 ms)
2015-02-23 05:12:01 +1100: slow request (1585 ms)
2015-02-23 05:12:03 +1100: slow request (1002 ms)
2015-02-23 05:12:15 +1100: 60 successful reqs
2015-02-23 05:12:27 +1100: slow request (1443 ms)
2015-02-23 05:12:30 +1100: slow request (1231 ms)
2015-02-23 05:12:42 +1100: slow request (1293 ms)
2015-02-23 05:12:48 +1100: slow request (2101 ms)
2015-02-23 05:12:50 +1100: slow request (1418 ms)
2015-02-23 05:13:26 +1100: slow request (1238 ms)
2015-02-23 05:13:29 +1100: slow request (1907 ms)
2015-02-23 05:13:31 +1100: slow request (1802 ms)
2015-02-23 05:13:52 +1100: slow request (1801 ms)
2015-02-23 05:13:54 +1100: slow request (1894 ms)
2015-02-23 05:14:02 +1100: 60 successful reqs
2015-02-23 05:14:14 +1100: slow request (1163 ms)
2015-02-23 05:14:16 +1100: slow request (1203 ms)
2015-02-23 05:15:39 +1100: 60 successful reqs
2015-02-23 05:16:54 +1100: slow request (1697 ms)
2015-02-23 05:16:57 +1100: slow request (1703 ms)
2015-02-23 05:17:04 +1100: slow request (1077 ms)
2015-02-23 05:17:19 +1100: 60 successful reqs
2015-02-23 05:17:58 +1100: slow request (1630 ms)
2015-02-23 05:18:00 +1100: slow request (1186 ms)
2015-02-23 05:18:11 +1100: slow request (1624 ms)
2015-02-23 05:18:13 +1100: slow request (1125 ms)
2015-02-23 05:18:58 +1100: 60 successful reqs
2015-02-23 05:20:34 +1100: 60 successful reqs
2015-02-23 05:22:10 +1100: 60 successful reqs
2015-02-23 05:23:45 +1100: 60 successful reqs
2015-02-23 05:25:22 +1100: 60 successful reqs
2015-02-23 05:26:58 +1100: 60 successful reqs
2015-02-23 05:28:34 +1100: 60 successful reqs
2015-02-23 05:30:12 +1100: 60 successful reqs
2015-02-23 05:31:47 +1100: 60 successful reqs
2015-02-23 05:33:23 +1100: 60 successful reqs
2015-02-23 05:34:58 +1100: 60 successful reqs
2015-02-23 05:36:34 +1100: 60 successful reqs
2015-02-23 05:38:09 +1100: 60 successful reqs
2015-02-23 05:39:44 +1100: 60 successful reqs
2015-02-23 05:41:20 +1100: 60 successful reqs
2015-02-23 05:42:55 +1100: 60 successful reqs
2015-02-23 05:43:01 +1100: slow request (1071 ms)
2015-02-23 05:43:05 +1100: slow request (1118 ms)
2015-02-23 05:43:09 +1100: slow request (1388 ms)
2015-02-23 05:43:11 +1100: slow request (1172 ms)
2015-02-23 05:44:24 +1100: slow request (1298 ms)
2015-02-23 05:44:26 +1100: slow request (1188 ms)
2015-02-23 05:44:36 +1100: 60 successful reqs
2015-02-23 05:46:12 +1100: 60 successful reqs
2015-02-23 05:47:04 +1100: slow request (1056 ms)
2015-02-23 05:47:20 +1100: slow request (1325 ms)
2015-02-23 05:47:22 +1100: slow request (1532 ms)
2015-02-23 05:47:50 +1100: 60 successful reqs
2015-02-23 05:49:26 +1100: 60 successful reqs
2015-02-23 05:51:01 +1100: 60 successful reqs
2015-02-23 05:52:37 +1100: 60 successful reqs
2015-02-23 05:54:13 +1100: 60 successful reqs
2015-02-23 05:55:48 +1100: 60 successful reqs
2015-02-23 05:57:23 +1100: 60 successful reqs
2015-02-23 05:58:59 +1100: 60 successful reqs
2015-02-23 06:00:35 +1100: 60 successful reqs
2015-02-23 06:02:10 +1100: 60 successful reqs
2015-02-23 06:03:45 +1100: 60 successful reqs
2015-02-23 06:04:44 +1100: slow request (1767 ms)
2015-02-23 06:04:47 +1100: slow request (1261 ms)
2015-02-23 06:05:23 +1100: 60 successful reqs
2015-02-23 06:06:58 +1100: 60 successful reqs
2015-02-23 06:08:33 +1100: 60 successful reqs
2015-02-23 06:10:09 +1100: 60 successful reqs
2015-02-23 06:11:45 +1100: 60 successful reqs
2015-02-23 06:13:21 +1100: 60 successful reqs
2015-02-23 06:14:56 +1100: 60 successful reqs
2015-02-23 06:16:32 +1100: 60 successful reqs
2015-02-23 06:18:08 +1100: 60 successful reqs
2015-02-23 06:19:43 +1100: 60 successful reqs
2015-02-23 06:21:19 +1100: 60 successful reqs
2015-02-23 06:22:55 +1100: 60 successful reqs
2015-02-23 06:24:30 +1100: 60 successful reqs
2015-02-23 06:26:07 +1100: 60 successful reqs
2015-02-23 06:27:42 +1100: 60 successful reqs
2015-02-23 06:29:17 +1100: 60 successful reqs
2015-02-23 06:30:53 +1100: 60 successful reqs
2015-02-23 06:32:28 +1100: 60 successful reqs
2015-02-23 06:34:03 +1100: 60 successful reqs
2015-02-23 06:35:39 +1100: 60 successful reqs
2015-02-23 06:37:14 +1100: 60 successful reqs
2015-02-23 06:38:49 +1100: 60 successful reqs
2015-02-23 06:40:25 +1100: 60 successful reqs
2015-02-23 06:42:00 +1100: 60 successful reqs
2015-02-23 06:43:36 +1100: 60 successful reqs
2015-02-23 06:45:11 +1100: 60 successful reqs
2015-02-23 06:46:46 +1100: 60 successful reqs
2015-02-23 06:48:21 +1100: 60 successful reqs
2015-02-23 06:49:56 +1100: 60 successful reqs
2015-02-23 06:51:32 +1100: 60 successful reqs
2015-02-23 06:53:07 +1100: 60 successful reqs
2015-02-23 06:54:43 +1100: 60 successful reqs
2015-02-23 06:56:18 +1100: 60 successful reqs
2015-02-23 06:57:54 +1100: 60 successful reqs
2015-02-23 06:59:30 +1100: 60 successful reqs
2015-02-23 07:01:05 +1100: 60 successful reqs
2015-02-23 07:01:27 +1100: slow request (1067 ms)
2015-02-23 07:02:41 +1100: 60 successful reqs
2015-02-23 07:04:16 +1100: 60 successful reqs
2015-02-23 07:05:51 +1100: 60 successful reqs
2015-02-23 07:07:26 +1100: 60 successful reqs
2015-02-23 07:09:01 +1100: 60 successful reqs
2015-02-23 07:10:36 +1100: 60 successful reqs
2015-02-23 07:12:12 +1100: 60 successful reqs
2015-02-23 07:13:47 +1100: 60 successful reqs
2015-02-23 07:15:23 +1100: 60 successful reqs
2015-02-23 07:16:58 +1100: 60 successful reqs
2015-02-23 07:18:34 +1100: 60 successful reqs
2015-02-23 07:20:00 +1100: slow request (2485 ms)
2015-02-23 07:20:11 +1100: 60 successful reqs
2015-02-23 07:21:46 +1100: 60 successful reqs
2015-02-23 07:23:21 +1100: 60 successful reqs
2015-02-23 07:24:57 +1100: 60 successful reqs
2015-02-23 07:26:32 +1100: 60 successful reqs
2015-02-23 07:28:08 +1100: 60 successful reqs
2015-02-23 07:29:44 +1100: 60 successful reqs
2015-02-23 07:31:19 +1100: 60 successful reqs
2015-02-23 07:32:55 +1100: 60 successful reqs
2015-02-23 07:34:30 +1100: 60 successful reqs
2015-02-23 07:36:06 +1100: 60 successful reqs
2015-02-23 07:37:41 +1100: 60 successful reqs
2015-02-23 07:39:16 +1100: 60 successful reqs
2015-02-23 07:40:52 +1100: 60 successful reqs
2015-02-23 07:42:27 +1100: 60 successful reqs
2015-02-23 07:44:03 +1100: 60 successful reqs
2015-02-23 07:44:41 +1100: slow request (1056 ms)
2015-02-23 07:45:39 +1100: 60 successful reqs
2015-02-23 07:47:15 +1100: 60 successful reqs
2015-02-23 07:48:33 +1100: slow request (1108 ms)
2015-02-23 07:48:52 +1100: 60 successful reqs
2015-02-23 07:50:27 +1100: 60 successful reqs
2015-02-23 07:52:03 +1100: 60 successful reqs
2015-02-23 07:53:39 +1100: 60 successful reqs
2015-02-23 07:54:52 +1100: slow request (1018 ms)
2015-02-23 07:55:15 +1100: 60 successful reqs
2015-02-23 07:56:50 +1100: 60 successful reqs
2015-02-23 07:58:26 +1100: 60 successful reqs
2015-02-23 08:00:01 +1100: 60 successful reqs
2015-02-23 08:01:37 +1100: 60 successful reqs
2015-02-23 08:03:12 +1100: 60 successful reqs
2015-02-23 08:04:47 +1100: 60 successful reqs
2015-02-23 08:06:23 +1100: 60 successful reqs
2015-02-23 08:07:58 +1100: 60 successful reqs
2015-02-23 08:09:33 +1100: 60 successful reqs
2015-02-23 08:09:57 +1100: slow request (1053 ms)
2015-02-23 08:11:10 +1100: 60 successful reqs
2015-02-23 08:12:46 +1100: 60 successful reqs
2015-02-23 08:14:21 +1100: 60 successful reqs
2015-02-23 08:15:57 +1100: 60 successful reqs
2015-02-23 08:17:32 +1100: 60 successful reqs
2015-02-23 08:17:44 +1100: slow request (1834 ms)
2015-02-23 08:17:46 +1100: slow request (1248 ms)
2015-02-23 08:18:58 +1100: slow request (1278 ms)
2015-02-23 08:19:10 +1100: 60 successful reqs
2015-02-23 08:20:46 +1100: 60 successful reqs
2015-02-23 08:20:56 +1100: slow request (1383 ms)
2015-02-23 08:21:18 +1100: slow request (1381 ms)
2015-02-23 08:21:23 +1100: slow request (1954 ms)
2015-02-23 08:21:32 +1100: slow request (1135 ms)
2015-02-23 08:21:56 +1100: slow request (1192 ms)
2015-02-23 08:22:28 +1100: 60 successful reqs
2015-02-23 08:22:31 +1100: slow request (1197 ms)
2015-02-23 08:22:33 +1100: slow request (1159 ms)
2015-02-23 08:22:36 +1100: slow request (1303 ms)
2015-02-23 08:23:24 +1100: slow request (1560 ms)
2015-02-23 08:23:27 +1100: slow request (2321 ms)
2015-02-23 08:23:30 +1100: slow request (1471 ms)
2015-02-23 08:23:34 +1100: slow request (1216 ms)
2015-02-23 08:24:09 +1100: slow request (1043 ms)
2015-02-23 08:24:11 +1100: 60 successful reqs
2015-02-23 08:24:24 +1100: slow request (1030 ms)
2015-02-23 08:24:33 +1100: slow request (1611 ms)
2015-02-23 08:24:35 +1100: slow request (1322 ms)
2015-02-23 08:24:38 +1100: slow request (1768 ms)
2015-02-23 08:25:07 +1100: slow request (1010 ms)
2015-02-23 08:25:14 +1100: slow request (1173 ms)
2015-02-23 08:25:53 +1100: slow request (1092 ms)
2015-02-23 08:25:53 +1100: 60 successful reqs
2015-02-23 08:26:42 +1100: slow request (1198 ms)
2015-02-23 08:27:31 +1100: 60 successful reqs
2015-02-23 08:28:39 +1100: slow request (1247 ms)
2015-02-23 08:29:07 +1100: 60 successful reqs
2015-02-23 08:30:44 +1100: 60 successful reqs
2015-02-23 08:32:22 +1100: 60 successful reqs
2015-02-23 08:33:57 +1100: 60 successful reqs
2015-02-23 08:35:17 +1100: slow request (1102 ms)
2015-02-23 08:35:20 +1100: slow request (1528 ms)
2015-02-23 08:35:22 +1100: slow request (1349 ms)
2015-02-23 08:35:24 +1100: slow request (1032 ms)
2015-02-23 08:35:29 +1100: slow request (1663 ms)
2015-02-23 08:35:37 +1100: 60 successful reqs
2015-02-23 08:37:12 +1100: 60 successful reqs
2015-02-23 08:38:48 +1100: 60 successful reqs
2015-02-23 08:39:26 +1100: slow request (1399 ms)
2015-02-23 08:39:29 +1100: slow request (1491 ms)
2015-02-23 08:40:27 +1100: 60 successful reqs
2015-02-23 08:42:08 +1100: 60 successful reqs
2015-02-23 08:43:48 +1100: 60 successful reqs
2015-02-23 08:45:29 +1100: 60 successful reqs
2015-02-23 08:45:52 +1100: slow request (1315 ms)
2015-02-23 08:47:10 +1100: 60 successful reqs
2015-02-23 08:47:16 +1100: slow request (1425 ms)
2015-02-23 08:47:19 +1100: slow request (1930 ms)
2015-02-23 08:48:52 +1100: 60 successful reqs
2015-02-23 08:50:31 +1100: 60 successful reqs
2015-02-23 08:51:19 +1100: slow request (1005 ms)
2015-02-23 08:52:08 +1100: 60 successful reqs
2015-02-23 08:53:37 +1100: slow request (1085 ms)
2015-02-23 08:53:45 +1100: 60 successful reqs
2015-02-23 08:53:50 +1100: slow request (1125 ms)
2015-02-23 08:54:19 +1100: slow request (1341 ms)
2015-02-23 08:54:31 +1100: slow request (1259 ms)
2015-02-23 08:54:35 +1100: slow request (2291 ms)
2015-02-23 08:54:37 +1100: slow request (1629 ms)
2015-02-23 08:54:39 +1100: slow request (1159 ms)
2015-02-23 08:55:27 +1100: 60 successful reqs
2015-02-23 08:57:04 +1100: 60 successful reqs
2015-02-23 08:58:40 +1100: 60 successful reqs
2015-02-23 08:59:54 +1100: slow request (1257 ms)
2015-02-23 08:59:56 +1100: slow request (1187 ms)
2015-02-23 09:00:19 +1100: 60 successful reqs
2015-02-23 09:01:54 +1100: 60 successful reqs
2015-02-23 09:03:30 +1100: 60 successful reqs
2015-02-23 09:05:05 +1100: 60 successful reqs
2015-02-23 09:06:41 +1100: 60 successful reqs

zero outages, 2 checks that took 5 seconds and a handful of checks taking 1.5-2 seconds.

SQL logs are happy as well. I don't know if this is "fixed" but stuff is way better that it was now.

[boomzilla]

@sam said:

except for 2^13,

Hmm...yes, that seemed to happen right after a post.