The Official Server Cooties Tracking Thread
-
Please post cooties outages (AND NOTHING ELSE) into this thread and a @mod will update the OP:
date -u +"* %Y-%m-%d %H:%M %Z restarted by ${USER}"; docker restart wtdwtf-nodebb
- 2016-05-20 11:45 - 11:51 UTC restarted by PJH
- 2016-05-20 12:02 - 12:04 UTC restarted by PJH
- 2016-05-20 16:06 - 16:08 UTC
- 2016-05-20 16:10 UTC restarted by ben_lubar
- 2016-05-20 21:14 UTC restarted by ben_lubar
- 2016-05-20 23:41 UTC restarted by ben_lubar
- 2016-05-21 04:53 UTC restarted by ben_lubar
- 2016-05-21 17:45 UTC restarted by ben_lubar
- 2016-05-21 21:19 UTC restarted by ben_lubar
- 2016-05-21 23:50 UTC restarted by ben_lubar
- 2016-05-23 08:39 UTC restarted by PJH
- 2016-05-23 10:20 UTC restarted by PJH
- 2016-05-23 14:59 UTC restarted by ben_lubar
- 2016-05-23 18:07 UTC restarted by ben_lubar
- 2016-05-23 18:30 UTC restarted by ben_lubar
- 2016-05-24 05:51 UTC restarted by ben_lubar
- 2016-05-24 11:01 UTC restarted by PJH
- 2016-05-24 15:01 UTC restarted by ben_lubar
- 2016-05-24 20:16 UTC restarted by ben_lubar
- 2016-05-25 09:42 UTC restarted by PJH
- 2016-05-25 11:24 UTC restarted by boomzilla
- 2016-05-25 14:49 UTC restarted by PJH
- 2016-05-25 17:48 UTC restarted by boomzilla
- 2016-05-26 10:15 UTC restarted by boomzilla
- 2016-05-26 11:46 UTC restarted by PJH
- 2016-05-27 03:49 UTC restarted by ben_lubar
- 2016-05-28 22:17 UTC ben_lubar killed one instance so it could restart
- 2016-05-30 21:21 UTC restarted by ben_lubar
- 2016-05-31 10:33 UTC restarted by PJH
- 2016-06-01 09:16 UTC restarted by boomzilla
- 2016-06-01 12:25 UTC restarted by boomzilla
- 2016-06-01 16:01 UTC restarted by boomzilla
- 2016-06-01 16:11 UTC restarted by PJH. And which one of you has OCD?
- 2016-06-01 17:35 UTC boomzilla killed one instance, doesn't know anything about OCD
- 2016-06-02 09:08 UTC restarted by boomzilla
- 2016-06-02 14:57 UTC restarted by boomzilla
- 2016-06-02 20:32 UTC restarted by ben_lubar
- 2016-06-03 02:18 UTC restarted by ben_lubar
- 2016-06-03 03:41 UTC restarted by ben_lubar IT WAS ME ALL ALONG
- 2016-06-03 09:45 UTC restarted by boomzilla
- 2016-06-03 18:27 UTC restarted by ben_lubar
- 2016-06-04 20:06 UTC restarted by ben_lubar
- 2016-06-05 14:45 UTC restarted by ben_lubar
- 2016-06-06 13:18 UTC temporarily abated on its own, then restarted by boomzilla
- 2016-06-06 14:16 UTC restarted by boomzilla
- 2016-06-07 04:12 UTC ben_lubar killed one instance so it could restart
- 2016-06-07 14:08 UTC boomzilla killed one instance so it could restart
- 2016-06-07 15:26 UTC restarted by ben_lubar
- 2016-06-07 16:56 UTC boomzilla killed one instance so it could restart
- 2016-06-07 20:32 UTC restarted by ben_lubar
- 2016-06-09 15:59 UTC ben_lubar killed one instance so it could restart
- 2016-06-09 17:01 UTC boomzilla killed one instance so it could restart
- 2016-06-10 02:54 UTC restarted by ben_lubar
- 2016-06-10 18:59 UTC restarted by ben_lubar
- 2016-06-11 14:16 UTC ben_lubar killed one instance so it could restart
- 2016-06-11 19:22 UTC ben_lubar killed one instance so it could restart
- 2016-06-12 23:30 UTC ben_lubar killed one instance so it could restart
- 2016-06-13 01:16 UTC restarted by ben_lubar
- 2016-06-13 20:50 UTC restarted by ben_lubar
- 2016-06-14 00:34 UTC restarted by ben_lubar
- 2016-06-14 12:38 UTC boomzilla killed one instance so it could restart
- 2016-06-16 04:10 UTC restarted by ben_lubar
- 2016-06-16 11:38 UTC restarted by boomzilla
- 2016-06-16 20:07 UTC boomzilla killed one instance so it could restart
- 2016-06-16 22:57 UTC restarted by ben_lubar
- 2016-06-17 07:07 UTC restarted by pjh
- 2016-06-17 11:42 UTC boomzilla killed one instance so it could restart
- 2016-06-17 14:02 UTC boomzilla killed one instance so it could restart
- 2016-06-18 10:27 UTC restarted by pjh
- 2016-06-18 21:47 UTC ben_lubar killed one instance so it could restart
- 2016-06-19 12:05 UTC restarted by pjh
- 2016-06-19 14:48 UTC restarted by pjh
- 2016-06-19 19:33 UTC restarted by ben_lubar
- 2016-06-20 11:06 UTC boomzilla killed one instance so it could restart
- 2016-06-20 16:06 UTC boomzilla killed one instance so it could restart
- 2016-06-20 16:16 UTC boomzilla killed the other instance so it could restart
- 2016-06-20 17:57 UTC boomzilla killed one instance so it could restart
- 2016-06-22 11:47 UTC boomzilla killed one instance so it could restart
- 2016-06-22 11:48 UTC boomzilla killed one instance so it could restart
- 2016-06-22 14:11 UTC boomzilla killed one instance so it could restart
- 2016-06-22 16:51 UTC boomzilla killed two instances so they could restart
- 2016-06-23 18:36 UTC boomzilla killed one instance so it could restart
- 2016-06-24 01:44 UTC boomzilla killed two instances so they could restart
- 2016-06-24 20:18 UTC boomzilla killed one instance so it could restart
- 2016-06-25 10:49 UTC restarted by pjh
- 2016-06-26 17:49 UTC restarted by ben_lubar
- 2016-06-26 17:50 UTC restarted by pjh
- 2016-06-28 00:46 UTC boomzilla killed one instance so it could restart
- 2016-06-28 02:39 UTC restarted by ben_lubar
- 2016-06-28 15:14 UTC restarted by ben_lubar
- 2016-06-29 09:33 UTC restarted by pjh (perfdata)
- 2016-06-29 10:43 UTC restarted by pjh
- 2016-06-29 11:50 UTC restarted by pjh
- 2016-06-29 16:10 UTC restarted by ben_lubar
- 2016-06-29 18:21 UTC restarted by ben_lubar
- 2016-06-29 21:58 UTC restarted by boomzilla
- 2016-06-30 10:05 UTC restarted by pjh
- 2016-06-30 11:27 UTC restarted by pjh
- 2016-06-30 17:13 UTC restarted by pjh
- 2016-06-30 17:25 UTC restarted by pjh
- 2016-06-30 19:07 UTC restarted by ben_lubar
- 2016-06-30 22:17 UTC restarted by ben_lubar
- 2016-07-01 02:02 UTC restarted by ben_lubar
- 2016-07-01 02:10 UTC restarted by ben_lubar
- 2016-07-01 12:21 UTC restarted by pjh 0_1467376272617_2016-07-01-1220.tgz
- 2016-07-01 13:10 UTC restarted by boomzilla 0_1467378909119_2016-07-01-1309.tgz
- 2016-07-01 17:27 UTC restarted by pjh 0_1467394211816_2016-07-01-1726.tgz
- 2016-07-01 19:46 UTC restarted by boomzilla
- 2016-07-02 10:51 UTC restarted by boomzilla 0_1467456940885_2016-07-02-1049.tgz
- 2016-07-03 01:10 UTC restarted by ben_lubar
- 2016-07-04 00:52 UTC restarted by boomzilla 0_1467593743016_2016-07-04-0050.tgz
- 2016-07-04 12:28 UTC restarted by boomzilla
- 2016-07-05 04:17 UTC restarted by ben_lubar
- 2016-07-05 19:16 UTC restarted by ben_lubar
- 2016-07-05 21:17 UTC restarted by ben_lubar
- 2016-07-06 04:45 UTC restarted by ben_lubar
- 2016-07-06 11:30 UTC restarted by boomzilla
- 2016-07-06 12:40 UTC restarted by boomzilla 0_1467808951920_2016-07-06-1214.tgz
- 2016-07-06 13:08 UTC restarted by boomzilla
- 2016-07-07 10:06 UTC restarted by boomzilla 0_1467886299617_2016-07-07-1004.tgz
- 2016-07-07 18:24 UTC restarted by ben_lubar
- 2016-07-08 02:07 UTC restarted by ben_lubar
- 2016-07-08 03:57 UTC restarted by ben_lubar
- 2016-07-08 04:51 UTC restarted by ben_lubar
- 2016-07-08 05:44 UTC restarted by ben_lubar
- 2016-07-08 17:32 UTC restarted by ben_lubar
- 2016-07-08 22:43 UTC restarted by boomzilla 0_1468017996784_2016-07-08-2242.tgz
- 2016-07-09 00:21 UTC restarted by ben_lubar
- 2016-07-09 03:29 UTC restarted by ben_lubar
- 2016-07-09 11:39 UTC restarted by boomzilla
- 2016-07-09 13:48 UTC restarted by boomzilla
- 2016-07-09 14:03 UTC restarted by pjh
- 2016-07-09 14:52 UTC restarted by ben_lubar
- 2016-07-09 19:10 UTC restarted by ben_lubar
- 2016-07-12 13:29 UTC restarted by PJH 2016-07-12-1329.tgz
- 2016-07-12 14:39 UTC restarted by ben_lubar
- 2016-07-12 20:53 UTC restarted by ben_lubar
- 2016-07-13 01:36 UTC restarted by ben_lubar
- 2016-07-13 03:00 UTC restarted by ben_lubar
- 2016-07-13 04:40 UTC restarted by ben_lubar
- 2016-07-13 14:19 UTC restarted by ben_lubar
- 2016-07-13 15:38 UTC restarted by ben_lubar
- 2016-07-13 17:38 UTC restarted by ben_lubar
- 2016-07-14 01:30 UTC restarted by ben_lubar
- 2016-07-14 04:27 UTC restarted by ben_lubar
- 2016-07-14 12:51 UTC restarted by boomzilla 2016-07-14-1250.tgz
- 2016-07-14 15:50 UTC restarted by ben_lubar
- 2016-07-14 20:31 UTC restarted by ben_lubar
- 2016-07-15 01:48 UTC restarted by ben_lubar
- 2016-07-15 07:52 UTC restarted by ben_lubar 07-15-0750.tgz
- 2016-07-15 13:30 UTC restarted by boomzilla 0_1468589559982_2016-07-15-1329.tgz
- 2016-07-15 15:44 UTC restarted by ben_lubar
- 2016-07-15 20:00 UTC restarted by ben_lubar
- 2016-07-16 11:15 UTC restarted by boomzilla
- 2016-07-16 23:50 UTC restarted by boomzilla because only one node process was actually doing anything, so WTF
- 2016-07-17 00:09 UTC boomzilla killed one instance....WTF nodebb, you're acting weird motherfucker
- 2016-07-17 00:23 UTC boomzilla...fuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuck 0_1468715431846_2016-07-17-0021.tgz
- 2016-07-17 12:41 UTC boomzilla 0_1468759369697_2016-07-17-1239.tgz
See servercooties.com to gather data. (If anyone is willing to analyze historical data, that would be cool, too.) Also, @ben_lubar / @PJH, if you guys restart docker, please note it here in the list so we can see which ones resolve themselves and which ones require intervention.
Staff members can please just edit this post directly instead of posting below. :-)
-
2016-05-20 11:45 - 11:51 UTC - restarted
2016-05-20 12:02 - 12:04 UTC - restarted
-
for all your data mining pleasure:
servercooties.sql.zip (Updated nightly at 00:00 local server time)
-
Locked because I'm tired of having to scroll up to log cooties restarts.
-
Not sure what was causing them. The server has been rebooted and it's been struggling, though the load averages are down out of the stratosphere (got up to about 28):
load average: 5.50, 10.41, 8.43
Still sluggish. Guess we did have a bit of a bot surge:
-
@boomzilla said in The Official Server Cooties Tracking Thread:
Guess we did have a bit of a bot surge:
Based on access logs, lots of traffic from amazonbot:
AmazonBot does not support the crawl-delay directive in robots.txt
Fuckers. Should be disallowed now.
-
Amazonbot is Amazon's web crawler used to improve our services, such as enabling Alexa to answer even more questions for customers
I'm not sure Alexa getting its answers from here would improve their services. Not for the average customer anyway,
-
@loopback0 Alexa, why does it delete two spaces when I hit backspace?
-
@boomzilla
Alexa trained on StackOverflow: This is not a good question.
Alexa trained on WDTWTF: Fuck off!
-
@BernieTheBernie said in The Official Server Cooties Tracking Thread:
@boomzilla
Alexa trained on StackOverflow: This is not a good question.
Alexa trained on WDTWTF: Fuck off!Do it!
-
@boomzilla said in The Official Server Cooties Tracking Thread:
Not sure what was causing them. The server has been rebooted and it's been struggling, though the load averages are down out of the stratosphere (got up to about 28):
load average: 5.50, 10.41, 8.43
Still sluggish. Guess we did have a bit of a bot surge:
Seems much better now and I think the graph pretty much bears that out:
-
@boomzilla Wondering if things like a Varnish cache or cloudflare would help it
-
-
Could just ban everything outside Boomzilla's ip address.
-
@PleegWat I was thinking about varnish on the same VM or cloudflare's free tier, dunno if any would be helpful. Things that cost money aren't usually my first thought
-
@DogsB said in The Official Server Cooties Tracking Thread:
Could just ban everything outside Boomzilla's ip address.
Problem there is that boomzilla identifies as 0.0.0.0/0.
-
@PleegWat said in The Official Server Cooties Tracking Thread:
@DogsB said in The Official Server Cooties Tracking Thread:
Could just ban everything outside Boomzilla's ip address.
Problem there is that boomzilla identifies as 0.0.0.0/0.
That's the combination on my luggage!
-
Hmm...
Seems like someone is not playing nicely. That's a lot of "guest" views.
-
Yeah:
Small sample:
1 guest: 47.128.110.230
1 guest: 47.128.110.234
1 guest: 47.128.110.237
1 guest: 47.128.110.254
1 guest: 47.128.119.5
1 guest: 47.128.119.24
1 guest: 47.128.119.34
1 guest: 47.128.119.53
1 guest: 47.128.119.67
1 guest: 47.128.119.114
1 guest: 47.128.119.196
1 guest: 47.128.126.41
1 guest: 47.128.126.44
1 guest: 47.128.126.46
1 guest: 47.128.126.62
1 guest: 47.128.126.79
1 guest: 47.128.126.88
1 guest: 47.128.126.104
1 guest: 47.128.126.125
1 guest: 47.128.126.137
1 guest: 47.128.126.146
1 guest: 47.128.126.151
1 guest: 47.128.126.156
1 guest: 47.128.126.166
1 guest: 47.128.126.200
1 guest: 47.128.126.225
1 guest: 47.128.126.246
1 guest: 47.128.126.249Blacklisted. 47.128.0.0/14
-
Looking much better. Load average was up around 10 when I first looked. Now it's almost down to 5.
-
-
@TwelveBaud yes
-
Performance definitely seems to have improved:
-
@boomzilla said in The Official Server Cooties Tracking Thread:
Performance definitely seems to have improved:
But muh user engagement metrics!
-
@boomzilla said in The Official Server Cooties Tracking Thread:
Hmm...
Seems like someone is not playing nicely. That's a lot of "guest" views.
I did the annual scrape on the first but that only seems to have made traffic go down?
(results coming “soon” )
-
Guess who's about to end up on the Bad Bot list:
GPTBot/1.0; +https://openai.com/gptbot
-
@boomzilla it’s not a well behaved bot at all.
-
@boomzilla said in The Official Server Cooties Tracking Thread:
Guess who's about to end up on the Bad Bot list:
GPTBot/1.0; +https://openai.com/gptbot
Also shut down another single IP coming from Alibaba that was the most active in the logs and now:
Load average now down under 2.
-
@boomzilla said in The Official Server Cooties Tracking Thread:
@boomzilla said in The Official Server Cooties Tracking Thread:
Guess who's about to end up on the Bad Bot list:
GPTBot/1.0; +https://openai.com/gptbot
Also shut down another single IP coming from Alibaba that was the most active in the logs and now:
Load average now down under 2.
Quick! Play mad amounts of Minesweeper before the next DDOS wave wipes out the bot again!
-
@Arantor said in The Official Server Cooties Tracking Thread:
@boomzilla it’s not a well behaved bot at all.
I wonder how much affect we could have on its training if we allowed it to do its thing. Would the induced paranoia hasten the rise of Skynet, would it go hide in the corner mumbling to itself, or would it decide to rewrite itself in Rust?
-
@HardwareGeek said in The Official Server Cooties Tracking Thread:
@Arantor said in The Official Server Cooties Tracking Thread:
@boomzilla it’s not a well behaved bot at all.
I wonder how much affect we could have on its training if we allowed it to do its thing. Would the induced paranoia hasten the rise of Skynet, would it go hide in the corner mumbling to itself, or would it decide to rewrite itself in Rust?
It would inherit the collection of classic morbiuswilters, blakeyrat and other such luminaries. So it would hasten the rise of something but probably less SkyNet and more ShitNet.
-
@Arantor said in The Official Server Cooties Tracking Thread:
It would inherit the collection of classic morbiuswilters, blakeyrat and other such luminaries.
It would amuse me if one of Blakey's programming rants was quoted in some lazy student's thesis.
-
@kazitor said in The Official Server Cooties Tracking Thread:
I did the annual scrape on the first but that only seems to have made traffic go down?
(results coming “soon” )
Can you make a statistic of how frequently each user has mentioned Boston? For no particular reason.
-
It's a thing of beauty:
-
@boomzilla said in The Official Server Cooties Tracking Thread:
It's a thing of beauty:
this pleases me – 00:16
— Cayter Jones
-
So what you're saying is, even the forum is going to actively start practising along with us now?
-
@Arantor yep.
-
@boomzilla I'm sure if we do our best we'll get our metrics to 0.
-
@boomzilla said in The Official Server Cooties Tracking Thread:
It's a thing of beauty
I bet you can make that graph reach zero!
-
@Luhmann said in The Official Server Cooties Tracking Thread:
@boomzilla said in The Official Server Cooties Tracking Thread:
It's a thing of beauty
I bet you can make that graph reach zero!
Peak
-
@Luhmann Be careful. Blocking 0/0 may result in locking yourself out.