Servercooties are annoying again


  • Winner of the 2016 Presidential Election

    So, I think this is my second time make a topic like this.
    But the server is down like waaaaaay too much again.

    Is /t/1000 killing it? Are we importing again? Is somebody installing new versions of things?

    @sam @codinghorror @pjh

    Filed Under: Will this topic be created? The suspense is killing me


  • Discourse touched me in a no-no place

    I've no idea what's going on - just noticed it was down and restarted the docker instance to get it back up again, but a quick grep through the logs isn't showing anything obvious..


  • ♿ (Parody)

    Sidekiq threw some errors about using too much memory. Then there seemed to be errors relating to too many DB connections being used.


  • BINNED

    Any thought to re-trying the import given that the site is barely usable right now anyway?



  • Does not seem to have had any impact yet.

    something's REALLY broken (ie. restart won't fix it)


  • Discourse touched me in a no-no place

    Yup... and I don't know enough about Discourse to figure out what it is...


  • FoxDev

    something seems to have settled down.....


  • Discourse touched me in a no-no place

    Nothing I've done - last thing I did was around 1600 on that graph from memory.


  • FoxDev

    -shrug-

    maybe @sam poked something?


  • Discourse touched me in a no-no place

    Possibly - someone's generated a website report out of schedule, it wasn't me and I don't think it was @boomzilla. Sam's the only other who would be interested in it.


  • Banned

    Yeah ... me

    This is weird:

    Today:

    [Anonymous]      204360.32 111647 - (POST)(84579.11) -(48333.03) topics/show(27718.85) user_avatars/show(17144.66)  
                                      user_avatars/show_letter(7538.84)                                                
    
    

    vs Yesterday

    [Anonymous]     22950.56 99529 topics/show(12267.28) user_avatars/show(3749.92)                                 
                                   user_avatars/show_letter(1900.33) list/latest(1240.70) -(853.66)                 
    

    WTF is anonymous doing POSTing shit, we need to look at the actual reqs in the logs, can you dig in @pjh with grep.


  • Winner of the 2016 Presidential Election

    Just out of curiousity, what do those numbers on a normal user look like?
    Did they also rise from yesterday?

    Filed Under: you can publizise my data on this if thats an issue... not that I am normal, though


  • Banned

    Normal users and bots are stable, this is the only big anomaly, smells like a spam attack


  • Discourse touched me in a no-no place

    @sam said:

    can you dig in @pjh with grep.

    I'm not sure what I should be looking for that would be causing anonymous POSTing...


  • Banned

    These bots need to back down, the put wood on burning fire

    root@what:/var/discourse/shared/standalone/log/var-log/nginx# cat access.log | grep POST | grep -v message-bus | grep '"-"'
    
    [18/Aug/2015:16:54:07 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:RaceProUK)" "-" 500 276 "-" 0.019 0.219 "-"
    [18/Aug/2015:16:54:08 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:NetBot)" "-" 500 276 "-" 0.024 0.221 "-"
    [18/Aug/2015:16:54:10 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.013 0.056 "-"
    [18/Aug/2015:16:54:10 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.017 0.061 "-"
    [18/Aug/2015:16:54:10 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.014 0.059 "-"
    [18/Aug/2015:16:54:10 +0000] 81.246.111.66 "POST /topics/timings HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" "-" 500 72 "https://what.thedailywtf.com/t/anonymo
    us-posting/47642" 0.015 0.015 "-"
    [18/Aug/2015:16:54:10 +0000] 54.77.156.15 "POST /session HTTP/1.1" "SockBot 0.13.0 \x22Devious Daine\x22 @codinghorrorbot" "-" 500 281 "-" 0.013 0.132 "-"
    [18/Aug/2015:16:54:13 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.031 0.079 "-"
    [18/Aug/2015:16:54:14 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.029 0.073 "-"
    [18/Aug/2015:16:54:14 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.023 0.072 "-"
    [18/Aug/2015:16:54:14 +0000] 81.246.111.66 "POST /topics/timings HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" "-" 500 71 "https://what.thedailywtf.com/t/anonymo
    us-posting/47642" 0.018 0.018 "-"
    [18/Aug/2015:16:54:15 +0000] 81.246.111.66 "POST /topics/timings HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" "-" 500 71 "https://what.thedailywtf.com/t/anonymo
    us-posting/47642" 0.029 0.029 "-"
    [18/Aug/2015:16:54:17 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.023 0.064 "-"
    [18/Aug/2015:16:54:18 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.040 0.086 "-"
    [18/Aug/2015:16:54:18 +0000] 54.77.156.15 "POST /session HTTP/1.1" "SockBot 0.13.0 \x22Devious Daine\x22 @codinghorrorbot" "-" 500 281 "-" 0.027 0.157 "-"
    [18/Aug/2015:16:54:18 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.016 0.065 "-"
    [18/Aug/2015:16:54:19 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:RaceProUK)" "-" 500 276 "-" 0.014 0.199 "-"
    [18/Aug/2015:16:54:19 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:NetBot)" "-" 500 276 "-" 0.036 0.221 "-"
    [18/Aug/2015:16:54:20 +0000] 188.242.4.6 "POST /draft.json HTTP/1.1" "Mozilla/5.0 (Linux; Android 5.0; SM-N9005 Build/LRX21V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.133 Mobile Safari/537.
    36" "-" 500 76 "https://what.thedailywtf.com/t/the-bad-jokes-topic/48503/751" 0.037 0.037 "-"
    [18/Aug/2015:16:54:21 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.028 0.073 "-"
    [18/Aug/2015:16:54:21 +0000] 188.242.4.6 "POST /topics/timings HTTP/1.1" "Mozilla/5.0 (Linux; Android 5.0; SM-N9005 Build/LRX21V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.133 Mobile Safari/
    537.36" "-" 500 73 "https://what.thedailywtf.com/t/the-bad-jokes-topic/48503/751" 0.030 0.030 "-"
    [18/Aug/2015:16:54:22 +0000] 188.242.4.6 "POST /draft.json HTTP/1.1" "Mozilla/5.0 (Linux; Android 5.0; SM-N9005 Build/LRX21V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.133 Mobile Safari/537.
    36" "-" 500 74 "https://what.thedailywtf.com/t/the-bad-jokes-topic/48503/751" 0.040 0.040 "-"
    [18/Aug/2015:16:54:22 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.035 0.090 "-"
    [18/Aug/2015:16:54:22 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.036 0.082 "-"
    [18/Aug/2015:16:54:25 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.017 0.060 "-"
    [18/Aug/2015:16:54:25 +0000] 54.77.156.15 "POST /session HTTP/1.1" "SockBot 0.13.0 \x22Devious Daine\x22 @codinghorrorbot" "-" 500 281 "-" 0.013 0.132 "-"
    [18/Aug/2015:16:54:26 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.041 0.085 "-"
    [18/Aug/2015:16:54:26 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.344 0.388 "-"
    [18/Aug/2015:16:54:29 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.023 0.073 "-"
    [18/Aug/2015:16:54:30 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:RaceProUK)" "-" 500 276 "-" 0.022 0.204 "-"
    [18/Aug/2015:16:54:30 +0000] 81.246.111.66 "POST /topics/timings HTTP/1.1" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0" "-" 500 75 "https://what.thedailywtf.com/t/anonymo
    us-posting/47642" 0.037 0.037 "-"
    [18/Aug/2015:16:54:30 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.029 0.073 "-"
    [18/Aug/2015:16:54:30 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:Zoidberg)" "-" 500 276 "-" 0.037 0.083 "-"
    [18/Aug/2015:16:54:31 +0000] 54.187.192.234 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:RaceProUK; user:NetBot)" "-" 500 276 "-" 0.028 0.216 "-"
    [18/Aug/2015:16:54:31 +0000] 188.242.4.6 "POST /draft.json HTTP/1.1" "Mozilla/5.0 (Linux; Android 5.0; SM-N9005 Build/LRX21V) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.133 Mobile Safari/537.
    36" "-" 500 76 "https://what.thedailywtf.com/t/the-bad-jokes-topic/48503/751" 0.025 0.025 "-"
    [18/Aug/2015:16:54:33 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:accalia)" "-" 500 276 "-" 0.067 0.099 "-"
    [18/Aug/2015:16:54:33 +0000] 54.77.156.15 "POST /session HTTP/1.1" "SockBot 0.13.0 \x22Devious Daine\x22 @codinghorrorbot" "-" 500 281 "-" 0.047 0.174 "-"
    [18/Aug/2015:16:54:34 +0000] 104.131.28.45 "POST /session HTTP/1.1" "SockBot/2.9.3 (Angelic Angora; owner:accalia; user:sockbot)" "-" 500 276 "-" 0.058 0.101 "-"
    

  • Banned

    If a bot gets a 500, back down for 1 minute


  • Winner of the 2016 Presidential Election

    paging @accalia for this, just in case she hasn't been notified!

    Filed Under: meh


  • FoxDev

    on it!

    We'll take care of that.


  • Discourse touched me in a no-no place

    What's the ETA on getting our CSS back?


  • Discourse touched me in a no-no place

    @dkf said:

    What's the ETA on getting our CSS back?

    ?

    I've still got mine...


  • FoxDev

    PR completed waiting for travis to do his thang and for peer review before releasing


  • FoxDev

    Aproved, merged, and released.



  • @PJH said:

    restarted the docker instance

    Can that be a public button?


  • FoxDev


  • Discourse touched me in a no-no place

    @xaade said:

    Can that be a public button?

    Only if they put a public restart button there if it doesn't restart for some reason...



  • I'm not running an import.

    I SSH'd in, but there doesn't seem to be any memory or disk pressure. We have about 25% of the disk free and 50% of the memory. The logs didn't seem very helpful when I glanced at them - there were a bunch of "I'm running this template for this page now" and a single "I have no idea how to render index.php so I'm returning a 404" that I saw.


  • Discourse touched me in a no-no place

    @ben_lubar said:

    We have about 25% of the disk free

    .. which is insufficient for backups....


  • FoxDev

    @PJH said:

    .. which is insufficient for backups....

    which is :wtf:


  • Discourse touched me in a no-no place

    @PJH said:

    I've still got mine...

    Ah. Artefact of doing a reload during the cootiestorm. Nevermind.



  • @dkf said:

    cootiestorm

    I'm trying to think of a way to work that into a game as a game mechanic, but it seems really skeksis out of context.


  • Discourse touched me in a no-no place

    @ben_lubar said:

    jbepiblei tolvudypai skeksis

    Well all knew it was coming, the day Ben would start just randomly banging on the keyboard.



  • Hey, Skeksis is a real thing that people talk about in English.


  • Discourse touched me in a no-no place

    @ben_lubar said:

    Skeksis

    The Dark Crystal was, indeed, a pretty good movie.



  • @ben_lubar said:

    Hey, Skeksis is a realfictional thing that annoying people talk about in English.

    askdfsa'fsad'ye eqw'hrqwh'eqwrh.



  • and it's doing it again...



  • Well, me and Blakey still have twitter.



  • 500 Internal Server Error
    If you are the administrator of this website, then please read this web application's log file and/or the web server's log file to find out what went wrong.

    Well, the hell was that?



  • same as yesterday.


  • Discourse touched me in a no-no place

    @Maciejasjmj said:

    Well, the hell was that?

    Cootiestorm. Looks like it was about an hour long or so.



  • Guess who isn't an administrator, and thus, doesn't need to see that page.

    Now guess who doesn't need a page specifically calling them out to know something is borked.

    Is that error page the DiscoDefault?



  • Yeah, but normally at least nginx seems to work to serve the better-looking 500 page, now it went full-on dead.



  • It seems sidekick is munching all the ram..



  • They should have gone with 16GB after all! :cheese:



  • 📯 :managed-hosting.asx: 📯



  • and again

    It's ridiculous. I'd have complained earlier, but disco was down...
    @pjh @sam


  • Banned

    9 postgres: discourse discourse [local] UPDATE waiting          
    postgres   140  4.6 17.3 1198864 702880 ?      Ss   22:00   0:11 postgres: discourse discourse [local] UPDATE waiting          
    postgres   149  5.2 15.6 1188900 633772 ?      Ss   22:00   0:13 postgres: discourse discourse [local] UPDATE waiting          
    postgres   170  3.6 16.6 1206216 672744 ?      Ss   22:00   0:09 postgres: discourse discourse [local] UPDATE waiting          
    root      3749  0.0  0.0  21316  1636 ?        Ss   22:01   0:00 /bin/bash
    postgres  5685 73.1 14.9 1187848 605508 ?      Rs   22:02   1:40 postgres: discourse discourse [local] SELECT                  
    postgres  9936  2.9 15.4 1200956 627332 ?      Ss   22:03   0:02 postgres: discourse discourse [local] UPDATE waiting          
    root      9954  0.0  0.0   7472   616 ?        S+   22:04   0:00 tail -f production.log
    postgres  9965  0.1  0.4 1183664 19332 ?       Ss   22:04   0:00 postgres: discourse discourse [local] UPDATE waiting          
    postgres  9974  0.1  0.3 1183288 15420 ?       Ss   22:04   0:00 postgres: discourse discourse [local] UPDATE waiting          
    postgres  9985  2.2 10.4 1184920 424600 ?      Ss   22:04   0:01 postgres: discourse discourse [local] UPDATE waiting          
    discour+ 10009  3.5  4.7 431508 192676 ?       Sl   22:04   0:01 unicorn worker[0] -E production -c config/unicorn.conf.rb     
    postgres 10015  1.4  9.8 1184784 399912 ?      Ss   22:04   0:00 postgres: discourse discourse [local] UPDATE waiting          
    discour+ 10032 17.5  5.1 1007488 208616 ?      Sl   22:04   0:03 unicorn worker[2] -E production -c config/unicorn.conf.rb     
    postgres 10038  2.2  3.9 1189932 161288 ?      Ss   22:04   0:00 postgres: discourse discourse [local] UPDATE waiting          
    root     10040  0.2  0.0  21316  1976 ?        Ss   22:04   0:00 /bin/bash
    discour+ 10306  4.3  4.6 425248 187648 ?       Sl   22:04   0:00 unicorn worker[1] -E production -c config/unicorn.conf.rb     
    postgres 10333  0.6  0.5 1183404 20412 ?       Ss   22:04   0:00 postgres: discourse discourse [local] UPDATE waiting          
    postgres 11855  1.2  4.8 1182368 196840 ?      Ss   22:04   0:00 postgres: discourse discourse [local] idle                    
    postgres 11856  1.6  5.7 1183620 234692 ?      Ss   22:04   0:00 postgres: discourse discourse [local] idle                    
    discour+ 12439  6.2  4.8 439700 197900 ?       Sl   22:04   0:00 unicorn worker[3] -E production -c config/unicorn.conf.rb     
    postgres 12445  1.0  1.1 1185084 46000 ?       Ss   22:04   0:00 postgres: discourse discourse [local] UPDATE waiting  
    

  • Banned

    DB is totally blocked on something .... looking


  • Banned

    Jobs::PurgeInactive has been failing quite a lot and its running now


  • Banned

    While I figure out why this scheduled task is choking the db up.

    Cooties gone


  • FoxDev

    This post is deleted!

Log in to reply