Servercooties are annoying again
-
So, I think this is my second time make a topic like this.
But the server is down like waaaaaay too much again.Is /t/1000 killing it? Are we importing again? Is somebody installing new versions of things?
Filed Under: Will this topic be created? The suspense is killing me
-
I've no idea what's going on - just noticed it was down and restarted the docker instance to get it back up again, but a quick grep through the logs isn't showing anything obvious..
-
Sidekiq threw some errors about using too much memory. Then there seemed to be errors relating to too many DB connections being used.
-
Any thought to re-trying the import given that the site is barely usable right now anyway?
-
Does not seem to have had any impact yet.
something's REALLY broken (ie. restart won't fix it)
-
Yup... and I don't know enough about Discourse to figure out what it is...
-
something seems to have settled down.....
-
Nothing I've done - last thing I did was around 1600 on that graph from memory.
-
-shrug-
maybe @sam poked something?
-
Possibly - someone's generated a website report out of schedule, it wasn't me and I don't think it was @boomzilla. Sam's the only other who would be interested in it.
-
Yeah ... me
This is weird:
Today:
[Anonymous] 204360.32 111647 - (POST)(84579.11) -(48333.03) topics/show(27718.85) user_avatars/show(17144.66) user_avatars/show_letter(7538.84)
vs Yesterday
[Anonymous] 22950.56 99529 topics/show(12267.28) user_avatars/show(3749.92) user_avatars/show_letter(1900.33) list/latest(1240.70) -(853.66)
WTF is anonymous doing POSTing shit, we need to look at the actual reqs in the logs, can you dig in @pjh with grep.
-
Just out of curiousity, what do those numbers on a normal user look like?
Did they also rise from yesterday?Filed Under: you can publizise my data on this if thats an issue... not that I am normal, though
-
Normal users and bots are stable, this is the only big anomaly, smells like a spam attack
-
can you dig in @pjh with grep.
I'm not sure what I should be looking for that would be causing anonymous POSTing...
-
These bots need to back down, the put wood on burning fire
root@what:/var/discourse/shared/standalone/log/var-log/nginx# cat access.log | grep POST | grep -v message-bus | grep '"-"'
[18/Aug/2015:16:54:07 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:RaceProUK)" "-" 500 276 "-" 0.019 0.219 "-" [18/Aug/2015:16:54:08 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:NetBot)" "-" 500 276 "-" 0.024 0.221 "-" [18/Aug/2015:16:54:10 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.013 0.056 "-" [18/Aug/2015:16:54:10 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.017 0.061 "-" [18/Aug/2015:16:54:10 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.014 0.059 "-" [18/Aug/2015:16:54:10 +0000] 81.246.111.66 "POST /topics/timings HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" "-" 500 72 "https://what.thedailywtf.com/t/anonymo us-posting/47642" 0.015 0.015 "-" [18/Aug/2015:16:54:10 +0000] 54.77.156.15 "POST /session HTTP/1.1" "SockBot 0.13.0 \x22Devious Daine\x22 @codinghorrorbot" "-" 500 281 "-" 0.013 0.132 "-" [18/Aug/2015:16:54:13 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.031 0.079 "-" [18/Aug/2015:16:54:14 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.029 0.073 "-" [18/Aug/2015:16:54:14 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.023 0.072 "-" [18/Aug/2015:16:54:14 +0000] 81.246.111.66 "POST /topics/timings HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" "-" 500 71 "https://what.thedailywtf.com/t/anonymo us-posting/47642" 0.018 0.018 "-" [18/Aug/2015:16:54:15 +0000] 81.246.111.66 "POST /topics/timings HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" "-" 500 71 "https://what.thedailywtf.com/t/anonymo us-posting/47642" 0.029 0.029 "-" [18/Aug/2015:16:54:17 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.023 0.064 "-" [18/Aug/2015:16:54:18 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.040 0.086 "-" [18/Aug/2015:16:54:18 +0000] 54.77.156.15 "POST /session HTTP/1.1" "SockBot 0.13.0 \x22Devious Daine\x22 @codinghorrorbot" "-" 500 281 "-" 0.027 0.157 "-" [18/Aug/2015:16:54:18 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.016 0.065 "-" [18/Aug/2015:16:54:19 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:RaceProUK)" "-" 500 276 "-" 0.014 0.199 "-" [18/Aug/2015:16:54:19 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:NetBot)" "-" 500 276 "-" 0.036 0.221 "-" [18/Aug/2015:16:54:20 +0000] 188.242.4.6 "POST /draft.json HTTP/1.1" "Mozilla/5.0 (Linux; Android 5.0; SM-N9005 Build/LRX21V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.133 Mobile Safari/537. 36" "-" 500 76 "https://what.thedailywtf.com/t/the-bad-jokes-topic/48503/751" 0.037 0.037 "-" [18/Aug/2015:16:54:21 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.028 0.073 "-" [18/Aug/2015:16:54:21 +0000] 188.242.4.6 "POST /topics/timings HTTP/1.1" "Mozilla/5.0 (Linux; Android 5.0; SM-N9005 Build/LRX21V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.133 Mobile Safari/ 537.36" "-" 500 73 "https://what.thedailywtf.com/t/the-bad-jokes-topic/48503/751" 0.030 0.030 "-" [18/Aug/2015:16:54:22 +0000] 188.242.4.6 "POST /draft.json HTTP/1.1" "Mozilla/5.0 (Linux; Android 5.0; SM-N9005 Build/LRX21V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.133 Mobile Safari/537. 36" "-" 500 74 "https://what.thedailywtf.com/t/the-bad-jokes-topic/48503/751" 0.040 0.040 "-" [18/Aug/2015:16:54:22 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.035 0.090 "-" [18/Aug/2015:16:54:22 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.036 0.082 "-" [18/Aug/2015:16:54:25 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.017 0.060 "-" [18/Aug/2015:16:54:25 +0000] 54.77.156.15 "POST /session HTTP/1.1" "SockBot 0.13.0 \x22Devious Daine\x22 @codinghorrorbot" "-" 500 281 "-" 0.013 0.132 "-" [18/Aug/2015:16:54:26 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.041 0.085 "-" [18/Aug/2015:16:54:26 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.344 0.388 "-" [18/Aug/2015:16:54:29 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.023 0.073 "-" [18/Aug/2015:16:54:30 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:RaceProUK)" "-" 500 276 "-" 0.022 0.204 "-" [18/Aug/2015:16:54:30 +0000] 81.246.111.66 "POST /topics/timings HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" "-" 500 75 "https://what.thedailywtf.com/t/anonymo us-posting/47642" 0.037 0.037 "-" [18/Aug/2015:16:54:30 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.029 0.073 "-" [18/Aug/2015:16:54:30 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.037 0.083 "-" [18/Aug/2015:16:54:31 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:NetBot)" "-" 500 276 "-" 0.028 0.216 "-" [18/Aug/2015:16:54:31 +0000] 188.242.4.6 "POST /draft.json HTTP/1.1" "Mozilla/5.0 (Linux; Android 5.0; SM-N9005 Build/LRX21V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.133 Mobile Safari/537. 36" "-" 500 76 "https://what.thedailywtf.com/t/the-bad-jokes-topic/48503/751" 0.025 0.025 "-" [18/Aug/2015:16:54:33 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.067 0.099 "-" [18/Aug/2015:16:54:33 +0000] 54.77.156.15 "POST /session HTTP/1.1" "SockBot 0.13.0 \x22Devious Daine\x22 @codinghorrorbot" "-" 500 281 "-" 0.047 0.174 "-" [18/Aug/2015:16:54:34 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.058 0.101 "-"
-
If a bot gets a 500, back down for 1 minute
-
-
on it!
We'll take care of that.
-
What's the ETA on getting our CSS back?
-
-
PR completed waiting for travis to do his thang and for peer review before releasing
-
Aproved, merged, and released.
-
-
Can that be a public button?
-
Can that be a public button?
Only if they put a public
restart
button there if it doesn't restart for some reason...
-
I'm not running an import.
I SSH'd in, but there doesn't seem to be any memory or disk pressure. We have about 25% of the disk free and 50% of the memory. The logs didn't seem very helpful when I glanced at them - there were a bunch of "I'm running this template for this page now" and a single "I have no idea how to render index.php so I'm returning a 404" that I saw.
-
-
-
-
cootiestorm
I'm trying to think of a way to work that into a game as a game mechanic, but it seems really skeksis out of context.
-
jbepiblei tolvudypai skeksis
Well all knew it was coming, the day Ben would start just randomly banging on the keyboard.
-
Hey, Skeksis is a real thing that people talk about in English.
-
-
Hey, Skeksis is a
realfictional thing that annoying people talk about in English.askdfsa'fsad'ye eqw'hrqwh'eqwrh.
-
and it's doing it again...
-
Well, me and Blakey still have twitter.
-
500 Internal Server Error
If you are the administrator of this website, then please read this web application's log file and/or the web server's log file to find out what went wrong.Well, the hell was that?
-
same as yesterday.
-
-
Guess who isn't an administrator, and thus, doesn't need to see that page.
Now guess who doesn't need a page specifically calling them out to know something is borked.
Is that error page the DiscoDefault?
-
Yeah, but normally at least nginx seems to work to serve the better-looking 500 page, now it went full-on dead.
-
It seems sidekick is munching all the ram..
-
They should have gone with 16GB after all!
-
📯 :managed-hosting.asx: 📯
-
-
9 postgres: discourse discourse [local] UPDATE waiting postgres 140 4.6 17.3 1198864 702880 ? Ss 22:00 0:11 postgres: discourse discourse [local] UPDATE waiting postgres 149 5.2 15.6 1188900 633772 ? Ss 22:00 0:13 postgres: discourse discourse [local] UPDATE waiting postgres 170 3.6 16.6 1206216 672744 ? Ss 22:00 0:09 postgres: discourse discourse [local] UPDATE waiting root 3749 0.0 0.0 21316 1636 ? Ss 22:01 0:00 /bin/bash postgres 5685 73.1 14.9 1187848 605508 ? Rs 22:02 1:40 postgres: discourse discourse [local] SELECT postgres 9936 2.9 15.4 1200956 627332 ? Ss 22:03 0:02 postgres: discourse discourse [local] UPDATE waiting root 9954 0.0 0.0 7472 616 ? S+ 22:04 0:00 tail -f production.log postgres 9965 0.1 0.4 1183664 19332 ? Ss 22:04 0:00 postgres: discourse discourse [local] UPDATE waiting postgres 9974 0.1 0.3 1183288 15420 ? Ss 22:04 0:00 postgres: discourse discourse [local] UPDATE waiting postgres 9985 2.2 10.4 1184920 424600 ? Ss 22:04 0:01 postgres: discourse discourse [local] UPDATE waiting discour+ 10009 3.5 4.7 431508 192676 ? Sl 22:04 0:01 unicorn worker[0] -E production -c config/unicorn.conf.rb postgres 10015 1.4 9.8 1184784 399912 ? Ss 22:04 0:00 postgres: discourse discourse [local] UPDATE waiting discour+ 10032 17.5 5.1 1007488 208616 ? Sl 22:04 0:03 unicorn worker[2] -E production -c config/unicorn.conf.rb postgres 10038 2.2 3.9 1189932 161288 ? Ss 22:04 0:00 postgres: discourse discourse [local] UPDATE waiting root 10040 0.2 0.0 21316 1976 ? Ss 22:04 0:00 /bin/bash discour+ 10306 4.3 4.6 425248 187648 ? Sl 22:04 0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb postgres 10333 0.6 0.5 1183404 20412 ? Ss 22:04 0:00 postgres: discourse discourse [local] UPDATE waiting postgres 11855 1.2 4.8 1182368 196840 ? Ss 22:04 0:00 postgres: discourse discourse [local] idle postgres 11856 1.6 5.7 1183620 234692 ? Ss 22:04 0:00 postgres: discourse discourse [local] idle discour+ 12439 6.2 4.8 439700 197900 ? Sl 22:04 0:00 unicorn worker[3] -E production -c config/unicorn.conf.rb postgres 12445 1.0 1.1 1185084 46000 ? Ss 22:04 0:00 postgres: discourse discourse [local] UPDATE waiting
-
DB is totally blocked on something .... looking
-
Jobs::PurgeInactive has been failing quite a lot and its running now
-
While I figure out why this scheduled task is choking the db up.
Cooties gone
-
This post is deleted!