Bots and cooties


  • :belt_onion:

    Continuing the discussion from Discodefinition QC:

    @Kuro said:

    Filed Under: Also, leave the bots out of this. Servercooties are bad enough as is. Thank you

    I just wanted to expand on this because it keeps coming up, whether as a joke or seriously. I am not sure if @Kuro was joking here or not, but I want to create a topic here so we have something to refer to if it comes up again.

    Disclaimer: I don't run any bots or "cyberparts" (bot logged in as my account doing stuff) nor have I contributed any code towards currently used bots. I have no vested interest in this other than trying to clarify things.

    Disclaimer: This is based on the current knowledge of what does cause cooties. There is always a possibility that there are unknown factors.

    What causes cooties:

    So far, we know that one of the cootie sources that is easily triggered and reproducible can be thought of to have connection to bots. That is loading and participating in a big enough thread. /t/1000 is the obvious thing to point to, but Status thread and Bad ideas thread are reaching high post counts as well. We have discovered that loading a topic causes getting IDs of all posts in that topic. On a technical level, your browser actually requests http://what.thedailywtf.com/t/<topic_id>.json. This is what every user does every time they enter a topic.

    Other things that we know can be problematic is badge queries. This is carefully balanced for by admins and is no longer a major issue.

    Other causes of server cooties which we know the reasons for were all, AFAIK, patched already.

    What non-user activity caused cooties?

    Well, bots actually did, indirectly, at one time. They generate extra likes in /t/1000. Before we handled the badges correctly (and there were some patches applied, IIRC), badge queries caused cooties. This is no longer a significant factor.

    Other than that: we done fucked up when building servercooties.com. Oh, the irony. We polled /t/1000 in the same way a user would. Meaning, we requested http://what.thedailywtf.com/t/1000.json. That caused a terrible cootie storm, but also helped us identify that this is in fact a problem we can replicate and measure it. So... half a win? This is no longer done, for obvious reasons. We do still poll the Discopædia thread occasionally to grab new Discodefinitions. We are careful about this and will turn it off the first time it causes any actual problems.

    Why bots don't cause server cooties?

    Disclaimer: I'm going off what I learned about Discourse myself and what @accalia told me / us about SockBot architecture. The source is no GitHub for anyone to verify. As far as I'm aware all independently written bots use the same basic principles.

    Because they don't "browse" the site. They listen on the message-bus. This is pretty much akin to you getting notifications. After that they do all actions they are programmed for without "loading" the topic. Meaning, a human user replying to or liking a post causes more server load than a bot doing the same, because a human user must first load the topic, possibly scroll through it and only then do whatever they intend to. The bots skip the whole middle of it and cut straight to liking / posting.

    Conclusion: Bots are putting less of a strain on the server than any single human user. Period.


  • Winner of the 2016 Presidential Election

    It was more or less a joke. Hence it was put in Filed Under. (Though on some posts the whole post is a joke and the actual message is in File Under so I can see how you could be confused).

    I had the option to reference blakeyrat hating bots (and other users not liking them) or going for the lighthearted servercooties-topic.

    One thing you are forgetting to mention in your post is really important, though:

    We have very little knowledge of how Discourse actually works.
    People thought botting would not cause the forum to go down, it did.
    People thought liking was a pretty safe action... it also brought the forum down.
    People thought using Discourse was a pretty safe action.... they have been taught a lesson.

    The fact of the matter is: Bots may as well do something weird again. And I am not blaming @accalia or anybody for that. Some of the things are really hard to predict (though the connect every 400ms or whatever in the beginning was kinda dumb :D ).

    Filed Under: I am not hating on bots here, btw. Also this is a serious Filed Under


  • sockdevs

    I would like to add to all of this that, despite what some users may choose to believe, the bots do nothing that users themselves don't do. In fact, the only reason the bots can do anything at all is because the bot-writers have reverse-engineered what the Discourse UI itself does.

    Which actually leads to a rather interesting conclusion: the Discourse UI is a bot, albeit one that's directed by a lump of flesh in a chair :laughing:

    Basically, at the end of the day, the main cause of server cooties is the myriad issues and inefficiencies in Discourse itself.


  • sockdevs

    @Kuro said:

    People thought botting would not cause the forum to go down, it did.

    Then we tamed them ;)


  • :belt_onion:

    It's OK, no ill will towards you. I was suspecting a joke TBH, but I used the chance to put it all in one concise post that won't be buried inside other discussions. I was just Doing It Right™ damn it!

    @Kuro said:

    One thing you are forgetting to mention in your post is really important, though:

    We have very little knowledge of how Discourse actually works.

    Yes, hence the disclaimers. But the things that bots really do are in the category of:

    @Kuro said:

    People thought liking was a pretty safe action... it also brought the forum down.

    Which is what people do anyway. And shit got patched, partly because the bots helped us find the issue. If we had thousands of bots maybe some additional issues would surface. But the same issues would surface with thousands of users. Hell, maybe we should offer the Discodevs this: we'll give them the tools to stress test the things in a controlled environment. And then we all get cake.


  • sockdevs

    @Onyx said:

    Doing It Right™

    How dare you! :stuck_out_tongue:
    @Onyx said:
    If we had thousands of bots maybe some additional issues would surface. But the same issues would surface with thousands of users.

    QFT
    @Onyx said:
    Hell, maybe we should offer the Discodevs this: we'll give them the tools to stress test the things in a controlled environment.

    The tools are on GitHub, and I can guarantee at least one Discodev already knows that. Maybe ;)
    @Onyx said:
    And then we all get cake.

    :yum:
    Just so long as we're not baked beforehand, that is…


  • Winner of the 2016 Presidential Election

    @RaceProUK said:

    Then we tamed them

    Urgh, I feel like everything I say tonight sounds agressive... and I had a pretty nice day and am pretty relaxed...

    But yeah, you guys did. You also did a pretty good job with the bots. But the main message of that post was that nobody expected things to go like they did with Discourse.

    Let's put a random example here:
    Since the bots are connecting on a very specific pattern (once every X seconds). Whats to tell you that Discourse does not go crazy with that behaviour after a while? I mean besides that it would already have happened. It's these little things that make bots "problematic". And it's those things nobody can really verify.

    Also: Let's be fair here: Some other Discourse instances seem to run okayisher than ours. So people look for the things that are different over here. (the bigger topics and the bots spring to mind :smile:)

    @Onyx said:

    I used the chance to put it all in one concise post that won't be buried inside other discussions

    That was a good idea :smiley: I had to defend myself, though!

    @Onyx said:

    And shit got patched, partly because the bots helped us find the issue

    I might sound like blakeyrat here, but thats pretty hardcore QA here. And it impacts the experience of all users negatively. Whitescreens of death are just dumb. Also: Yes, that was clearly Discourses fault. But it also highlights what I tried to say: Discourse might break because of reasons you can't fathom :smiley:

    @RaceProUK said:

    Just so long as we're not baked beforehand, that is…

    That post was already baked. You can ask a TL4 to rebake it, though!

    Filed Under: Swooosh (not a serious Filed Under)


  • sockdevs

    @Kuro said:

    Urgh, I feel like everything I say tonight sounds agressive... and I had a pretty nice day and am pretty relaxed...

    If I was annoyed with you, I wouldn't be using emoji :stuck_out_tongue_winking_eye:
    @Kuro said:
    Since the bots are connecting on a very specific pattern (once every X seconds).

    Same as your browser is doing when you're on this site ;)
    @Kuro said:
    DiscourseSoftware might break because of reasons you can't fathom

    FTFY
    @Kuro said:
    @RaceProUK said:
    Just so long as we're not baked beforehand, that is…

    That post was already baked. You can ask a TL4 to rebake it, though!

    I was going for a Portal reference, but that works too :laughing:


  • Winner of the 2016 Presidential Election

    @RaceProUK said:

    If I was annoyed with you

    I am annoyed with my own way of writing right now. And I am still using Emoji :smiley:

    @RaceProUK said:

    Same as your browser is doing when you're on this site

    Servercooties because of browsers? Where is the Bug-report at chrome? ;P

    @RaceProUK said:

    I was going for a Portal reference, but that works too

    I even got that reference but I felt like a Discourse-reference was in order :smiley:

    Filed Under: I am clearly derailing this topic now... I'll just stop for now until somebody mentions or quotes me ;P


  • :belt_onion:

    @Kuro said:

    Filed Under: I am clearly derailing this topic now... I'll just stop for now until somebody mentions or quotes me

    My topic even! You scoundrel!

    Seriously now, it's important we cleared the air. I wasn't aiming at any accusations nor wanted to call anyone out. Just wanted somewhere we can have some facts stated in it's own topic.


  • sockdevs

    @Onyx said:

    . We do still poll the Discopædia thread occasionally to grab new Discodefinitions.

    once every five hours at present. i'm thinking of making it a daily task.

    @Onyx said:

    The source is no GitHub for anyone to verify

    it's also on github. :-P

    @Onyx said:

    After that they do all actions they are programmed for without "loading" the topic.

    not entirely correct. notifications that appear in /notifications (blue curcle numbers) will trigger a topic load of the first page of the topic. this is to allow the bot to check various permissions of the topic (such as is the topic muted, or is it in a disallowed category, or is the topic in a topic owned by a user that has been set to be ignored)

    but this is done just as a real person would /t/topicid/1.json and is triggered by notifications. most bots do not get notifications often.

    @Kuro said:

    though the connect every 400ms or whatever in the beginning was kinda dumb

    I hide my face in shame that i once thought that was an acceptable idea...

    @Kuro said:

    Since the bots are connecting on a very specific pattern (once every X seconds).

    They used to and that was an issue. they're reactionary now. they poll message-bus just like every other browser and only do extra things when summoned. (or triggered via configuration)

    Now that i wrote SockBot to9 use message-bus i tried to make it as humanlike as possible to minimise differential stress.

    The bots also have custom user agent strings to make any abuises they may cause much easier to find in logs.


  • :belt_onion:

    @accalia said:

    it's also on github.

    /me moves a piece on his board

    @accalia said:

    Now that i wrote SockBot to9 use

    Counter check!

    @accalia said:

    not entirely correct.

    Corrections welcome :)


  • sockdevs

    @accalia said:

    once every five hours at present. i'm thinking of making it a daily task.

    May as well; definitions don't get added that frequently ;)


  • sockdevs

    well reloading the definitions also updates avatars. ;-)


  • sockdevs

    @accalia said:

    well reloading the definitions also updates avatars. ;-)

    True, but you're not anywhere near as bad w.r.t. changing avatars as you used to be :stuck_out_tongue:


  • sockdevs

    @RaceProUK said:

    True, but you're not anywhere near as bad w.r.t. changing avatars as you used to be

    touchet.

    and i do have a ticket that we may be trying to grab discodefinitions too quickly and causing a minicooties storm on restart. i should make that change at the same time.



  • @accalia said:

    touchéet

    Ftfy


  • sockdevs

    I'm pretty sure that one was deliberate :stuck_out_tongue:


  • sockdevs

    @RaceProUK said:

    I'm pretty sure that one was deliberate

    guilty as charged. :-P



  • I think someone missed the joke.

    No, I'm not trying to start a topic or whatever with this.



  • @Kuro said:

    So people look for the things that are different over here. (the bigger topics and the bots spring to mind )

    • I suspect we mix a lot of weird unicode stuff more than most. I believe they have stuff in East Asian languages out there, so maybe less than those guys, but I know we've helped them find some ruby / pg related unicode things.
    • Obviously, we stress their retarded markdown stuff more than probably anyone.
    • We have a lot of auto-closing topics.
    • I'm sure we have more custom badge stuff.

  • Discourse touched me in a no-no place

    @Kuro said:

    People thought using Discourse was a pretty safe action.... they have been taught a lesson.

    Are you sure about that last part? I mean, we're all still here... :trollface:



  • @boomzilla said:

    I suspect we mix a lot of weird unicode stuff more than most.

    This was one of the downfalls of my bot code. Python and Unicode go together about as well as Ted Kennedy and drunk driving.



  • @mott555 said:

    about as well as Ted Kennedy and drunk driving.

    A Perfect Match?



  • @Luhmann said:

    A Perfect Match?

    More like excessive collateral damage.



  • @mott555 said:

    This was one of the downfalls of my bot code. Python and Unicode go together about as well as Ted Kennedy and drunk driving.

    Sounds like I'll be going Python 3 if I ever do the Discobot thing.


  • sockdevs

    to be fair i've had a fair few problems with sockbot on that score too.

    JS handles unicode better once it's in JS but parsing external text file in the first place needs.... persuasion... on occasion.


  • sockdevs

    @accalia said:

    text file sin

    Damn that sinning text file! Damn it straight to hell! :stuck_out_tongue:


  • sockdevs

    even freshly rested and caffeinated, and even actively trying not to @accalia i @accalia..... :facepalm:

    :-P



  • @Onyx said:

    If we had thousands of bots maybe some additional issues would surface. But the same issues would surface with thousands of users. Hell, maybe we should offer the Discodevs this: we'll give them the tools to stress test the things in a controlled environment.

    Is the UpperDiscoDev still in denial about the feasibility of bots on Discourse or has he been struck by reality already?


  • :belt_onion:

    I believe that, currently, bots are possible but they darken the heart of Discourse itself just by existing. The shadow is currently spreading over the Internet. We might need to call upon the great hero of security by obscurity soon, if this keeps going.


  • Winner of the 2016 Presidential Election

    /t/1/totallynottheposts.json


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.