The Official Status Thread
-
There is a lot of bot activity
I wouldn't have thought the load from the bots is that high compared to activity from the flesh-and-blood users; I'm sure we have far more people than bots active here. Plus, as I understand it, the bots tap into the same messagebus stuff that the website itself uses to live-update the topic lists and notifications.
-
Status: Thinking it's an hour later than it actually is
-
Status: Wishing it was an hour later than it actually is; I'd be an hour closer to WEEKEND!!
-
Status: Only 27 minutes to go until weekend.
-
There is a lot of bot activity. Need to figure out what impact bots vs people have. Getting live analytics on patterns is a tricky problem that I would love to solve.
I'm mostly certain mine were crashed out anyway, but just in case I powered down the VM that hosts my bots.
-
Ok, can we run a controlled experiment here then, please?
Can we have all bots shut down for a day or two this weekend? I'm pretty sure the biggest spikes happened every day at roughly the same time (@accalia, can we get a publicly accessible dump somewhere perhaps? Maybe we should make and expose a daily backup copy on servercooties.com?).
At least we can narrow it down then. And again, kill the bots. It's either their fault, in which case we'll have no problems and can start figuring it out, or it isn't and we'll eliminate that as a possible cause.
-
i'll kill mine for the weekend.
accalia@SockBot:~$ SockStatus SockBotAccalia start/running, process 28474 SockBotSockPuppet stop/waiting SockBot stop/waiting SockBotSystern stop/waiting SockBotTCotCDCK stop/waiting SockBotTodo stop/waiting SockBotTranslator stop/waiting SockBotZoidberg stop/waiting accalia@SockBot:~$there you go. that's mine down. for the weekend.
i posted the log from servercooties.com earlier and i'll have a monthly graph on plot.ly by end of weekend.
-
i posted the log from servercooties.com earlier
Missed it then. Will look up, I want to play with it, too.
i'll have a monthly graph on plot.ly by end of weekend
-
Missed it then. Will look up, I want to play with it, too.
i thinmk it was in the likes thread.... i'll save you teh searching.
oh... nope. not the likes thread.
-
Ok, can we run a controlled experiment here then, please?
Can we have all bots shut down for a day or two this weekend? I'm pretty sure the biggest spikes happened every day at roughly the same time (@accalia, can we get a publicly accessible dump somewhere perhaps? Maybe we should make and expose a daily backup copy on servercooties.com?).
At least we can narrow it down then. And again, kill the bots. It's either their fault, in which case we'll have no problems and can start figuring it out, or it isn't and we'll eliminate that as a possible cause.
…and that's mine down, though I doubt they're the issue. The major slowdowns seem to be around mid-afternoon UTC; of my bots, only NetBot binges, and only at 2330UTC. The rest of the time, they're just polling MessageBus/Notifications, except my cyberparts, which autolike. I'm pretty sure I personally create more traffic than all my bots combined
A thought occurs… wouldn't it make more sense to run this experiment during the week? Weekends, the traffic load on the site is a lot lower, and there are fewer performance issues.
-
Look, people didn't understand how tiny invisible critters could possibly be the cause of their explosive diarrhea, but wouldn't you know?
I'm inclined to side with your opinion, but we of all people should know better than assume hitting random buttons and cargo-culting will help us. Controlled experiments are the way to get shit done. And turning the bots off is probably the simplest and least intrusive first step we can take.
-
I've turned mine off; what more do you want?
I still think it's better to run this experiment mid-week; the data will be the ultimate arbiter, but it does seem to be during the week we get issues. Not that they never happen at weekends, but they're generally rarer and less serious.
-
Status
SyntaxError: /home/ubuntu/workspace/SockSite/config.json: Unexpected end of input at Object.parse (native) at Object.Module._extensions..json (module.js:482:27) at Module.load (module.js:356:32) at Function.Module._load (module.js:312:12) at Module.require (module.js:364:17) at require (module.js:380:17) at Object.<anonymous> (/home/ubuntu/workspace/SockSite/cache.js:7:14) at Module._compile (module.js:456:26) at Object.Module._extensions..js (module.js:474:10) at Module.load (module.js:356:32)uhh..... WTF? that is a valid JSON file!
-
I've turned mine off; what more do you want?
YOUR SOUL!
I still think it's better to run this experiment mid-week; the data will be the ultimate arbiter, but it does seem to be during the week we get issues.
Possible. But I will contemplate that in the daytime. It is 3AM after all.
-
YOUR SOUL!
Sure, but you'll need to sign the 2,048-page contract agreement first.Oh, and fight @accalia for it too
-
YOUR SOUL!
back of the line bub. i got here first.
uhh..... WTF? that is a valid JSON file!
nope it wasn't. my editor was lying to me. it said it was operating on the file on disk, but the file on disk was 0 bytes....
stupid lying editor.... stupid fox for not checking that sooner..... stupid git for.... wait, not git was what alerted me to the problem by saying i had unstaged changes in that file i hadn't changed. SMART GIT!
-
Status: 1645, VNC viewer crashed. I interpreted that as a sign that the workday was over.
-
Controlled experiments are the way to get shit done. And turning the bots off is probably the simplest and least intrusive first step we can take.
Thought: your controlled experiment has a flaw. There are
twothree changes being put in place during the first phase of your experiment:- Weekend usage is naturally reduced.
- Shutting off the bots will further reduce the load.
- Now that t/1000 has reached 40k, posting in that topic will slow down as well.
Any one of those factors could, in theory, cause an improvement in performance. If you were truly interested in performing a controlled experiment, you'd wait and see if things improved on Monday and Tuesday. If things did improve, then recent issues were likely due to the increased traffic in t/1000, which would be an indication that Discourse does not handle large, active topics well. If things did not improve through Tuesday, then try turning the bots off on say, Thursday, to see if that helped. Then, if turning the bots off improves things on Thursday/Friday, you have your answer, if not, you can continue your experiment into next weekend for the three factor experiment you are currently proposing.
As it is, your "controlled experiment" is not very controlled.
-
consider it a data point then. we'll run the weekend with them off then turn them back on for the week then turn them off for a tuesday/wednesday and compare notes. ;-)
with data you need more than one data point.
-
The weird thing is that not only does purple merge into itself, but black branches from and merges back into blue before blue gets another commit. And purple merges into blue twice.
-
The weird thing is that not only does purple merge into itself, but black branches from and merges back into blue before blue gets another commit. And purple merges into blue twice.
This is because
gitdoesn't actually store any actual information about what actually happens to your actual files, perhaps?
-
that's mine down
SockBotAccalia start/running, process 28474
Except the one that isn't
edit:
> sudo supervisorctl bender STOPPED Apr 17 01:03 AM chb STOPPED Apr 18 06:40 AM discobot_days STOPPED Apr 18 06:41 AM jfive STOPPED Apr 17 01:04 AM
-
I miss the Likes Thread mania already.
-
back of the line bub. i got here first.
Is it a long line? I'm not British, I don't really know how to queue.
SMART GIT!
Thanks!
Thought: your controlled experiment has a flaw.
Agreed, partially. While we were pretty active every time the slowdown hit we remained active even after it normalized. The amount of traffic doesn't seem to be the only factor. I'm interested to see what will happen with already reduced traffic and no bots. If we get any spikes they should be easier to analyse.
Also, I went and suggested stuff at 2AM. Badly formed ideas likely :P
-
Well would you look at that:
Let's see if it gets any worse. I wager that even if it doesn't those spikes are a factor not caused by increased activity, but rather just amplified by it.
Edit: need more coffee, those are small spikes... But why then... oh, read only!
Damn it.
-
-
Is it a long line?
You're the first to request my soul, that's f'sho; @accalia though is ahead by default :P
-
Well, yes, that cleared it.
But I did have posts refusing to be marked read, and I did catch a
BADstatus on servercooties, but I guess the averages diminished it's visibility on graph.This is weird. And not something I can deal with after I overslept and still didn't have my coffee.
-
Status:
The cooties are strong with this one today:
-
Tried to edit the typo away, got this:
-
Text copied from the popup:
SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data
-
That would mean it was empty? I guess?
More importantly, am I the only one who thinks Discourse is tilted in that screenshot? Zooming in would indicate not, but damn it, looks tilted to me!
Do I need more coffee? Please advise.
E_500count before successful posting: 5
-
Oh no, not broken at all
-
So, maybe we have an answer with the not-so-controlled experiment. Server cooties happen without bots involved, but that leaves a few other things to test.
-
Hmmm... did we run the servercooties website before the emails got fixed?
Where did I store that DB dump...
-
YES! WE DID!
Ok, I am now eagerly awaiting that graph so see if there's any correlation between emails getting fixed and server cooties...
-
Status: Sporadic 500 errors, but only when trying to view topics...
-
I figure I'm getting the same here, as I'm seeing the "failed to load topic" error and have to refresh to view topics again. Though I have to refresh a few times since it ends up giving me a blank page a few times.
500 error when posting, then refreshing never actually got past the blank page, I had to reload from my bookmark.
-
the cooties are strong with this one indeed....
what's gone wrong this time i wonder?
Attempt 6
-
Tried to "Reply as linked Topic" to a post. Continuing 500 error; never did get it to post. Went to the category in which I wanted to post, created a new topic, pasted the text from the failed post, and success.
-
Status:
Staring at the blue dot next to above post.
It's been a minute, Discourse, I read it.
Oh, it's gone now. Ok.
Status:
Trying to post this.
-
Ok, I am now eagerly awaiting that graph so see if there's any correlation between emails getting fixed and server cooties...
Are you suggesting that the email digests could be making the site unusable?
DAMN YOU SERVER BELGIUM ERRORS! GET THE FUCK OUT OF MY WAY!
WHY THE HELL DID I NEED TO TRY POSTING 22 TIME?
-
Status: we are now officially on the beta channel here, you can use the web interface to update from now @PJH
Anyone remember this? That was five days ago...
-
Status: Counting how many times Discourse has shit itself so badly it's been completely restarted today.
Current count: Two.
-
Tried to post in another topic. Tried over 200 times before I went off and did other stuff for a while. It seems to be better, for now, until next time.
-
Status: Counting how many times Discourse has shit itself so badly it's been completely restarted today.
Current count: Two.
Three.But that might have to do with Alex briefly turning the CS forum back on.
-
Status: Counting how many times Discourse has shit itself so badly it's been completely restarted today.
Current count: Two.
Four.Maybe; not sure I'm counting them right…
-
Status:
Wikipedia cannot convey the awesome. Even if the article was longer.
Oh look, onebox is dead, too. Well, it tried, appeared for a second and just gave up.
-
that seems tasty.
i wonder if this city has a croatian restaurant....
EDIT: found one! putting it in the list of places to go next time i'm in downtown
-
can be either cooked or baked
…baking is cooking…
Zagorski štrukli - Wikipedia
