Moar Cooties
-
Whoa.
Analyzed: /var/log/nginx/access.log,/var/log/nginx/access.log.1 on what.thedailywtf.com-app Limited to 2016-01-04T19:50:15+00:00 - 2016-01-05T19:50:15+00:00 ---------------------------------------------------------------------------------------------------- 05/Jan/2016:07:35:17 +0000 - 05/Jan/2016:07:35:16 +0000 ---------------------------------------------------------------------------------------------------- Top 30 users by Server Load Username Duration Reqs Routes -------- -------- ---- ------ [Anonymous] 82847.52 111061 topics/show(51280.14) user_avatars/show(10042.83) - (POST)(5372.43) -(4316.43) list/latest(3015.19) CodingHorrorBot 22246.49 13633 topics/show(18475.83) posts/show(3556.74) notifications/mark_read(125.29) notifications/index(88.63) discoursebot 21836.71 13519 topics/show(18056.28) posts/show(3601.23) notifications/index(107.92) notifications/mark_read(58.51) posts/create (POST)(12.77) PaulaBean 7242.00 11815 topics/show(7178.39) list/category_latest(56.74) topics/status(4.42) posts/create (POST)(2.44) accalia 3788.56 4467 topics/posts(1864.65) topics/timings (POST)(433.02) posts/create (POST)(324.21) list/latest(312.10) topics/show(276.11) Boner 3444.37 6058 topics/timings (POST)(2530.16) topics/show(231.65) topics/posts(200.56) user_avatars/show(68.11) users/show(62.53) loopback0 3233.36 5293 topics/timings (POST)(970.73) topics/show(911.63) posts/create (POST)(284.13) list/latest(242.43) topics/posts(163.00) Top 30 urls by Server Load Url Duration Reqs --- -------- ---- POST /topics/timings HTTP/1.1 46074.53 96516 GET /t/1000.json?include_raw=1&track_visit=true HTTP/1.1 36067.93 12441 POST /posts HTTP/1.1 5396.38 1145 POST /draft.json HTTP/1.1 3059.41 6539 GET / HTTP/1.1 2721.59 6289 POST /post_actions HTTP/1.1 2143.61 2127
-
Whoa.
I DIDN'T DO NOTHING!
I SWEAR!
i've looked at @loopback0's configs too, they're sane!
i don't know what's going on!
/me breaks down crying
why must the universe be so cruel!
</dramaqueen>
-
If that's not better tomorrow with a full day of updated bots... no fucking idea.
-
Can the bots easily be made to log the requests they make to a file or the console or something?
-
Can the bots easily be made to log the requests they make to a file or the console or something?
they do, to a certain degree already do you log those or just pipe to /dev/null?
the existing log should be enough to at least get me started in the right general direction of the issue
did you install via NPM or git?
of git what's the output of
git status
?
-
they do, to a certain degree already do you log those or just pipe to /dev/null?
I get these, which are left at the console seeing as they just run in a detached screen session.
[20:22:37] Polling Messages [20:23:14] Polling Messages [20:24:04] Polling Messages [20:25:18] Polling Messages [20:26:31] Polling Notifications
did you install via NPM or git?
Both?
Cloned it from git, used npm to install the dependencies.of git what's the output of git status?
(the untracked files are just the config files and the old versions of them)$ git status On branch master Your branch is up-to-date with 'origin/master'. Untracked files: (use "git add <file>..." to include in what will be committed) chb.conf.json chb2.bkup chb2.conf.json disco.conf.json nothing added to commit but untracked files present (use "git add" to track)
-
This would have included the time where @PJH saw fairly constant /t/1000 activity, so that's not a total surprise. Hopefully tomorrow's report will be better.
-
Are you sure you've only got one copy of the bots running? No old copy brought itself back from the dead?
-
I get these, which are left at the console seeing as they just run in a detached screen session.
[20:22:37] Polling Messages
[20:23:14] Polling Messages
[20:24:04] Polling Messages
[20:25:18] Polling Messages
[20:26:31] Polling Notificationsit's pretty chatty with those...
next time you start them can you tee that output to a text file for a day or so so i have some logs to look at?
Cloned it from git, used npm to install the dependencies.
hmm... that's a supported installation method....
I assume you are on master branch?
or did you clone a release tag?
-
Are you sure you've only got one copy of the bots running?
Fairly sure - I'm running them from the command prompt and detaching the screen session. I'm not sure they could be running multiple times. I'll double check.
-
next time you start them can you tee that output to a text file for a day or so so i have some logs to look at?
By the by, if you're running Node applications and want them to stay up consistantly, with nice features like auto-rotating logs, I highly recommend pm2: https://github.com/Unitech/pm2
I'm using it for @Vote right now, I like that at any point I can check on it by typing
pm2 logs mafiaBot
to see the latest logs.
-
Command invalid or no command issued. Try the
help
command.
-
I assume you are on master branch?
master.
next time you start them can you tee that output to a text file for a day or so so i have some logs to look at?
I can but it just logs the fact it made a request, rather than what that was.
-
I can
apreciate that.
but it just logs the fact it made a request, rather than what that was.
nah. it told you what the request was too.Polling Messages
means it's loading/message-bus/GUID/poll
Polling Notifications
means it's loading .... i forget which one that is but it's the JSON you get wehn you click on your avatar for your notifications.it has other messages it spits out when something happens.
such as when i mention @discoursebot it'll have different lines in there if you catch it before it scrolls beyond the reach of your scrollback buffer.
-
@accalia - Last Day Without A Discourse Bug: null
-
hi!
-
SockBot/2.11.9 (Cheery Chiffon; owner:loopback0; user:discoursebot)
I'm guessing now we know for sure he updated.
-
such as when i mention @discoursebot it'll have different lines in there if you catch it before it scrolls beyond the reach of your scrollback buffer.
[20:50:29] Polling Messages [20:50:30] Polling Messages [20:50:59] Polling Messages [20:51:31] Polling Messages [20:52:09] Polling Notifications [20:52:09] Polling Messages [20:52:10] Polling Messages [20:52:42] Polling Messages [20:53:20] Polling Messages [20:54:11] Polling Messages [20:54:12] Polling Notifications [20:54:14] Polling Messages [20:54:15] Polling Messages [20:54:16] replied notification #2984083 was not handled!
The last entry being when you replied to it.
-
The last entry being when you replied to it.
wait..... what?
.....ooookay.... apparetnly i unchattied the logs and forgot about it....
okay i'll think of something.
-
What about today's?
-
Today looks better for the bots. They're way down the list and /t/1000 doesn't appear at all.
-
Timing on cooties seems to have been consistent this week, around this time. How much history can we get out of servercooties.com? @accalia
-
Sorry, I just wanted to look at my own profile.
-
https://servercooties.com/raw?start=2016-01-01&end=2016-01-06
limit of 100k records per request, and it is liable to crash the site if you put the end date in the future.... not sure why.
obit the start and end dates and it will default to giving you the past 24 hours.
or you could poke about in the SQLite database (data from beginning of time* to about 0845 local when i pulled the backup)
servercooties.db.zip (43,228 KB)
here's a query to get you started.
SELECT p.key, c.status, c.responseTime, c.checkedAt FROM checks2 c JOIN pages p ON c.page = p.OID ORDER BY checkedAt ASC
* beginning of time being defined as when we started tracking this stuff
-
Thanks. I don't SQL but I can excel. I'll try to remember to use the US date format
-
US date format
ara ara?
those are ISO format dates!
they are big-endian dates because that is THE ONLY CORRECT WAY TO WRITE DATES!
-
-
-
fumblefingers
Wow, you're feeling pretty mean this morning, and worse, that insult doesn't even make sense in context!
-
Wow, you're feeling pretty mean this morning, and worse, that insult doesn't even make sense in context!
what? no! was explanation! not insult!
-
@FrostCat said:
Wow, you're feeling pretty mean this morning, and worse, that insult doesn't even make sense in context!
what? no! was explanation! not insult!It's a good thing I don't tease you about whooshes any more!
-
I'm more vanilla-bean. But no not yet a cyborg.
-
Spiky.
-
This is different:
-
You keep using that word. I do not think it means what you think it means.
ETA: 504 error trying to post this, and "Drafts offline."
-
I tried to like this but got 504 OK. Maybe posting works.
edit; and the like apparently worked too.
Saving the edit is a 504 OK though. I think Dischorse needs turning off and on again.
-
I tried to like this but got 504 OK.
I've seen more Teflon likes today than I have in the entire time since /t/1000 was disabled, I think.
-
What is going on with the site at the moment?
-
What is going on with the site at the moment?
Someone is bouncing on the corporate trampoline?
-
It definitely seems that way. I'm getting 504 and connection errors pretty much every time I try and go into a thread at the moment...
-
This is definitly the fault of that one topic with the probably short posts! Discourse wasn't designed to handle the overwhelming load of ~217 posts!
Jeff himself told us that nobody would play games on a forum anyway. So it's obvious this is the problem.
It doesn't show up in @boomzillas statistics because it is doing it so wrong that the service that tries to get the statistics crashes while gathering data!Filed Under: warned us! And we didn't listen! How foolish we were!
Also Filed Under:
-
Oooo That's new. 504 Gateway timeout from nginx...
-
-
I was meaning for this batch of cooties, not in general. There are some serious performance issues going on somewhere tonight.
-
Also ServerCooties.com™ is dead. @accalia.
Maybe we need isitjustmeorservercootiesdotcomcooties.com?
-
When did we go back to circles for avatars?
Wait, it is just this tab
-
Name: ec2-52-88-134-242.us-west-2.compute.amazonaws.com Address: 52.88.134.242
That looks familiar… now why does it look familiar…*fires up PuTTY*
*loads her SockBot config*…that would be my AWS instance
The load from that should have dropped significantly; while there are four bots there, they no longer look at t/1000. So combined they should make no more load than a single relatively active user. Actually, on that subject; @boomzilla, how's the load of my bots?
-
The load from that should have dropped significantly
Wasn't that near the top, it was just an example of what an AWS host name normally looks like.
-
What do you guys all use your bots for? I can see the use of one or two like shadowmod and votebot, but how many @Zoidberg's are there? Or do they have uses I'm just not aware of?
-
Is it me maybe?