Moar downtime? who was playing with bots again?


  • sockdevs

    seriously? who was playing with bots?

    anyone?


  • Winner of the 2016 Presidential Election

    I was just about to write a question like this in OneTopic but it wouldn't let me create it.

    Is this really an issue a bot can create, anyway? Or is the Server just being dumb again / are we running out of memory again?

    Paging @Sam because I seem to remember he wanted to take a look at our memory situation.

    Filed Under: Those responsible for this should be sacked | Those responsible for the sacking of those who should be sacked should also be sacked for good measure



  • I wonder if there is a cyclical job that runs at this time, or if it is just a USA lunch break usage spike?



  • @Kuro said:

    Is this really an issue a bot can create, anyway? Or is the Server just being dumb again / are we running out of memory again?

    The bots spend most of their time polling for notifications and notifications only, which is less polling then Discourse itself does when open in a browser. I can't see how they could be.
    The polling duration is easily changed if needed, though.


  • sockdevs

    we seem to be mostly back.

    i'm only 10% serious about it being the bots, we've brought this site to its knees more than once by accident.

    i monstly used that accusation because i thought it would be a funny topic title.

    but seriously. wth was that? @pjh, @sam? do we know?



  • I logged on to the server and it said out of memory again.So, then I shut it down and upgraded to 4GB.

    I blame Ruby for having such poor rubbish collection, and Linux for being generally rubbish.




  • sockdevs

    linux is hard to get right for sure, but frankly i'm shocked that we haven't been bringing a 2GB box to its knees more often on this forum.

    we are geeks and we love to push limits after all.



  • Given that I've run shitty-as-all-get-out Wordpress on a linux box with 2GB, I think we can place the blame squarely on Ruby. I used to have to use Redmine for issue tracking and the damned thing takes as much as 30 seconds just to show the login screen sometimes.



  • @apapadimoulis said:

    Linux for being generally rubbish.

    You had me up until there.



  • Command line:
      .\ffmpeg.exe -y -f concat -i concat.txt -b:v 5M -crf 10 -t 10 combined.webm -t 10 combined.mp4 2> ffmpeg.log
    
    
    Contents of concat.txt:
      ffconcat version 1.0
    
      file 'bay12games.gif'
      duration 2.20
    
      file 'toadyone.gif'
      duration 3.45
    
      file 'dwarf_fortress.gif'
      duration 18.00
    
      file 'wgen.gif'
      duration 195.18
    
      file '0000.gif'
      duration 3259.62
    
      file '0001.gif'
      duration 1898.24
    
    
    Contents of ffmpeg.log:
      ffmpeg version N-66278-g91459bd Copyright (c) 2000-2014 the FFmpeg developers
        built on Sep 14 2014 22:14:38 with gcc 4.8.3 (GCC)
        configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-decklink --enable-zlib
        libavutil      54.  7.100 / 54.  7.100
        libavcodec     56.  1.100 / 56.  1.100
        libavformat    56.  4.101 / 56.  4.101
        libavdevice    56.  0.100 / 56.  0.100
        libavfilter     5.  1.100 /  5.  1.100
        libswscale      3.  0.100 /  3.  0.100
        libswresample   1.  1.100 /  1.  1.100
        libpostproc    53.  0.100 / 53.  0.100
      Input #0, concat, from 'concat.txt':
        Duration: 01:29:36.69, start: 0.000000, bitrate: N/A
          Stream #0:0: Video: gif, bgra, 800x300, 50 fps, 50 tbr, 100 tbn, 100 tbc
      No pixel format specified, yuv444p for H.264 encoding chosen.
      Use -pix_fmt yuv420p for compatibility with outdated media players.
      [libvpx @ 00000000003880e0] v1.3.0
      [libx264 @ 0000000000388780] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
      [libx264 @ 0000000000388780] profile High 4:4:4 Predictive, level 3.1, 4:4:4 8-bit
      [libx264 @ 0000000000388780] 264 - core 142 r2479 dd79a61 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
      Output #0, webm, to 'combined.webm':
        Metadata:
          encoder         : Lavf56.4.101
          Stream #0:0: Video: vp8 (libvpx), yuva420p, 800x300, q=-1--1, 5000 kb/s, 50 fps, 1k tbn, 50 tbc
          Metadata:
            encoder         : Lavc56.1.100 libvpx
      Output #1, mp4, to 'combined.mp4':
        Metadata:
          encoder         : Lavf56.4.101
          Stream #1:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv444p, 800x300, q=-1--1, 50 fps, 12800 tbn, 50 tbc
          Metadata:
            encoder         : Lavc56.1.100 libx264
      Stream mapping:
        Stream #0:0 -> #0:0 (gif (native) -> vp8 (libvpx))
        Stream #0:0 -> #1:0 (gif (native) -> h264 (libx264))
      Press [q] to stop, [?] for help
      frame=   28 fps=0.0 q=0.0 q=0.0 size=      26kB time=00:00:00.56 bitrate= 385.3kbits/s    
      frame=   56 fps= 55 q=0.0 q=0.0 size=      41kB time=00:00:01.12 bitrate= 299.7kbits/s    
      frame=   85 fps= 56 q=0.0 q=31.0 size=      56kB time=00:00:01.70 bitrate= 270.1kbits/s    
      [concat @ 0000000004418b00] DTS -922328613750666 < 218 out of order
      DTS -922328613750666, next:2200000 st:0 invalid dropping
      PTS -922328613750666, next:2200000 invalid dropping st:0
      DTS -922328613750663, next:2230000 st:0 invalid dropping
      PTS -922328613750663, next:2230000 invalid dropping st:0
      frame=  112 fps= 55 q=0.0 q=31.0 size=      63kB time=00:00:02.26 bitrate= 227.5kbits/s dup=1 drop=0    
      DTS -922328613750660, next:2260000 st:0 invalid dropping
      PTS -922328613750660, next:2260000 invalid dropping st:0
      DTS -922328613750657, next:2290000 st:0 invalid dropping
      PTS -922328613750657, next:2290000 invalid dropping st:0
      DTS -922328613750654, next:2320000 st:0 invalid dropping
      PTS -922328613750654, next:2320000 invalid dropping st:0
      DTS -922328613750651, next:2350000 st:0 invalid dropping
      PTS -922328613750651, next:2350000 invalid dropping st:0
      DTS -922328613750648, next:2380000 st:0 invalid dropping
      PTS -922328613750648, next:2380000 invalid dropping st:0
      DTS -922328613750645, next:2410000 st:0 invalid dropping
      PTS -922328613750645, next:2410000 invalid dropping st:0
      DTS -922328613750642, next:2440000 st:0 invalid dropping
      PTS -922328613750642, next:2440000 invalid dropping st:0
      DTS -922328613750639, next:2470000 st:0 invalid dropping
      PTS -922328613750639, next:2470000 invalid dropping st:0
      DTS -922328613750636, next:2500000 st:0 invalid dropping
      PTS -922328613750636, next:2500000 invalid dropping st:0
      DTS -922328613750633, next:2530000 st:0 invalid dropping
      PTS -922328613750633, next:2530000 invalid dropping st:0
      DTS -922328613750630, next:2560000 st:0 invalid dropping
      PTS -922328613750630, next:2560000 invalid dropping st:0
      DTS -922328613750627, next:2590000 st:0 invalid dropping
      PTS -922328613750627, next:2590000 invalid dropping st:0
      DTS -922328613750624, next:2620000 st:0 invalid dropping
      PTS -922328613750624, next:2620000 invalid dropping st:0
      DTS -922328613750621, next:2650000 st:0 invalid dropping
      PTS -922328613750621, next:2650000 invalid dropping st:0
      DTS -922328613750618, next:2680000 st:0 invalid dropping
      PTS -922328613750618, next:2680000 invalid dropping st:0
      DTS -922328613750615, next:2710000 st:0 invalid dropping
      PTS -922328613750615, next:2710000 invalid dropping st:0
      DTS -922328613750612, next:2740000 st:0 invalid dropping
      PTS -922328613750612, next:2740000 invalid dropping st:0
      DTS -922328613750609, next:2770000 st:0 invalid dropping
      PTS -922328613750609, next:2770000 invalid dropping st:0
      DTS -922328613750606, next:2800000 st:0 invalid dropping
      PTS -922328613750606, next:2800000 invalid dropping st:0
      DTS -922328613750603, next:2830000 st:0 invalid dropping
      PTS -922328613750603, next:2830000 invalid dropping st:0
      DTS -922328613750600, next:2860000 st:0 invalid dropping
      PTS -922328613750600, next:2860000 invalid dropping st:0
      DTS -922328613750597, next:2890000 st:0 invalid dropping
      PTS -922328613750597, next:2890000 invalid dropping st:0
      DTS -922328613750594, next:2920000 st:0 invalid dropping
      PTS -922328613750594, next:2920000 invalid dropping st:0
      DTS -922328613750591, next:2950000 st:0 invalid dropping
      PTS -922328613750591, next:2950000 invalid dropping st:0
      DTS -922328613750588, next:2980000 st:0 invalid dropping
      PTS -922328613750588, next:2980000 invalid dropping st:0
      DTS -922328613750585, next:3010000 st:0 invalid dropping
      PTS -922328613750585, next:3010000 invalid dropping st:0
      DTS -922328613750582, next:3040000 st:0 invalid dropping
      PTS -922328613750582, next:3040000 invalid dropping st:0
      DTS -922328613750579, next:3070000 st:0 invalid dropping
      PTS -922328613750579, next:3070000 invalid dropping st:0
      frame=  140 fps= 55 q=0.0 q=31.0 size=      79kB time=00:00:03.10 bitrate= 207.9kbits/s dup=15 drop=0    
      DTS -922328613750576, next:3100000 st:0 invalid dropping
      PTS -922328613750576, next:3100000 invalid dropping st:0
      DTS -922328613750573, next:3130000 st:0 invalid dropping
      PTS -922328613750573, next:3130000 invalid dropping st:0
      DTS -922328613750570, next:3160000 st:0 invalid dropping
      PTS -922328613750570, next:3160000 invalid dropping st:0
      DTS -922328613750567, next:3190000 st:0 invalid dropping
      PTS -922328613750567, next:3190000 invalid dropping st:0
      DTS -922328613750564, next:3220000 st:0 invalid dropping
      PTS -922328613750564, next:3220000 invalid dropping st:0
      DTS -922328613750561, next:3250000 st:0 invalid dropping
      PTS -922328613750561, next:3250000 invalid dropping st:0
      DTS -922328613750558, next:3280000 st:0 invalid dropping
      PTS -922328613750558, next:3280000 invalid dropping st:0
      DTS -922328613750555, next:3310000 st:0 invalid dropping
      PTS -922328613750555, next:3310000 invalid dropping st:0
      DTS -922328613750552, next:3340000 st:0 invalid dropping
      PTS -922328613750552, next:3340000 invalid dropping st:0
      DTS -922328613750549, next:3370000 st:0 invalid dropping
      PTS -922328613750549, next:3370000 invalid dropping st:0
      DTS -922328613750546, next:3400000 st:0 invalid dropping
      PTS -922328613750546, next:3400000 invalid dropping st:0
      DTS -922328613750543, next:3430000 st:0 invalid dropping
      PTS -922328613750543, next:3430000 invalid dropping st:0
      DTS -922328613750540, next:3460000 st:0 invalid dropping
      PTS -922328613750540, next:3460000 invalid dropping st:0
      DTS -922328613750537, next:3490000 st:0 invalid dropping
      PTS -922328613750537, next:3490000 invalid dropping st:0
      DTS -922328613750534, next:3520000 st:0 invalid dropping
      PTS -922328613750534, next:3520000 invalid dropping st:0
      DTS -922328613750531, next:3550000 st:0 invalid dropping
      PTS -922328613750531, next:3550000 invalid dropping st:0
      DTS -922328613750528, next:3580000 st:0 invalid dropping
      PTS -922328613750528, next:3580000 invalid dropping st:0
      DTS -922328613750525, next:3610000 st:0 invalid dropping
      PTS -922328613750525, next:3610000 invalid dropping st:0
      DTS -922328613750522, next:3640000 st:0 invalid dropping
      PTS -922328613750522, next:3640000 invalid dropping st:0
      DTS -922328613750519, next:3670000 st:0 invalid dropping
      PTS -922328613750519, next:3670000 invalid dropping st:0
      DTS -922328613750516, next:3700000 st:0 invalid dropping
      PTS -922328613750516, next:3700000 invalid dropping st:0
      DTS -922328613750513, next:3730000 st:0 invalid dropping
      PTS -922328613750513, next:3730000 invalid dropping st:0
      DTS -922328613750510, next:3760000 st:0 invalid dropping
      PTS -922328613750510, next:3760000 invalid dropping st:0
      DTS -922328613750507, next:3790000 st:0 invalid dropping
      PTS -922328613750507, next:3790000 invalid dropping st:0
      DTS -922328613750504, next:3820000 st:0 invalid dropping
      PTS -922328613750504, next:3820000 invalid dropping st:0
      DTS -922328613750501, next:3850000 st:0 invalid dropping
      PTS -922328613750501, next:3850000 invalid dropping st:0
      DTS -922328613750498, next:3880000 st:0 invalid dropping
      PTS -922328613750498, next:3880000 invalid dropping st:0
      DTS -922328613750495, next:3910000 st:0 invalid dropping
      PTS -922328613750495, next:3910000 invalid dropping st:0
      frame=  168 fps= 55 q=0.0 q=31.0 size=      90kB time=00:00:03.94 bitrate= 186.2kbits/s dup=29 drop=0    
      DTS -922328613750492, next:3940000 st:0 invalid dropping
      PTS -922328613750492, next:3940000 invalid dropping st:0
      DTS -922328613750489, next:3970000 st:0 invalid dropping
      PTS -922328613750489, next:3970000 invalid dropping st:0
      DTS -922328613750486, next:4000000 st:0 invalid dropping
      PTS -922328613750486, next:4000000 invalid dropping st:0
      DTS -922328613750483, next:4030000 st:0 invalid dropping
      PTS -922328613750483, next:4030000 invalid dropping st:0
      DTS -922328613750480, next:4060000 st:0 invalid dropping
      PTS -922328613750480, next:4060000 invalid dropping st:0
      DTS -922328613750477, next:4090000 st:0 invalid dropping
      PTS -922328613750477, next:4090000 invalid dropping st:0
      DTS -922328613750474, next:4120000 st:0 invalid dropping
      PTS -922328613750474, next:4120000 invalid dropping st:0
      DTS -922328613750471, next:4150000 st:0 invalid dropping
      PTS -922328613750471, next:4150000 invalid dropping st:0
      DTS -922328613750468, next:4180000 st:0 invalid dropping
      PTS -922328613750468, next:4180000 invalid dropping st:0
      DTS -922328613750465, next:4210000 st:0 invalid dropping
      PTS -922328613750465, next:4210000 invalid dropping st:0
      DTS -922328613750462, next:4240000 st:0 invalid dropping
      PTS -922328613750462, next:4240000 invalid dropping st:0
      DTS -922328613750459, next:4270000 st:0 invalid dropping
      PTS -922328613750459, next:4270000 invalid dropping st:0
      DTS -922328613750456, next:4300000 st:0 invalid dropping
      PTS -922328613750456, next:4300000 invalid dropping st:0
      DTS -922328613750453, next:4330000 st:0 invalid dropping
      PTS -922328613750453, next:4330000 invalid dropping st:0
      DTS -922328613750450, next:4360000 st:0 invalid dropping
      PTS -922328613750450, next:4360000 invalid dropping st:0
      DTS -922328613750447, next:4390000 st:0 invalid dropping
      PTS -922328613750447, next:4390000 invalid dropping st:0
      DTS -922328613750444, next:4420000 st:0 invalid dropping
      PTS -922328613750444, next:4420000 invalid dropping st:0
      DTS -922328613750441, next:4450000 st:0 invalid dropping
      PTS -922328613750441, next:4450000 invalid dropping st:0
      DTS -922328613750438, next:4480000 st:0 invalid dropping
      PTS -922328613750438, next:4480000 invalid dropping st:0
      DTS -922328613750435, next:4510000 st:0 invalid dropping
      PTS -922328613750435, next:4510000 invalid dropping st:0
      DTS -922328613750432, next:4540000 st:0 invalid dropping
      PTS -922328613750432, next:4540000 invalid dropping st:0
      DTS -922328613750429, next:4570000 st:0 invalid dropping
      PTS -922328613750429, next:4570000 invalid dropping st:0
      DTS -922328613750426, next:4600000 st:0 invalid dropping
      PTS -922328613750426, next:4600000 invalid dropping st:0
      DTS -922328613750423, next:4630000 st:0 invalid dropping
      PTS -922328613750423, next:4630000 invalid dropping st:0
      DTS -922328613750420, next:4660000 st:0 invalid dropping
      PTS -922328613750420, next:4660000 invalid dropping st:0
      DTS -922328613750417, next:4690000 st:0 invalid dropping
      PTS -922328613750417, next:4690000 invalid dropping st:0
      DTS -922328613750414, next:4720000 st:0 invalid dropping
      PTS -922328613750414, next:4720000 invalid dropping st:0
      frame=  195 fps= 55 q=0.0 q=31.0 size=      98kB time=00:00:04.74 bitrate= 169.9kbits/s dup=42 drop=0    
      DTS -922328613750411, next:4750000 st:0 invalid dropping
      PTS -922328613750411, next:4750000 invalid dropping st:0
      DTS -922328613750408, next:4780000 st:0 invalid dropping
      PTS -922328613750408, next:4780000 invalid dropping st:0
      DTS -922328613750405, next:4810000 st:0 invalid dropping
      PTS -922328613750405, next:4810000 invalid dropping st:0
      DTS -922328613750402, next:4840000 st:0 invalid dropping
      PTS -922328613750402, next:4840000 invalid dropping st:0
      DTS -922328613750399, next:4870000 st:0 invalid dropping
      PTS -922328613750399, next:4870000 invalid dropping st:0
      DTS -922328613750396, next:4900000 st:0 invalid dropping
      PTS -922328613750396, next:4900000 invalid dropping st:0
      DTS -922328613750393, next:4930000 st:0 invalid dropping
      PTS -922328613750393, next:4930000 invalid dropping st:0
      DTS -922328613750390, next:4960000 st:0 invalid dropping
      PTS -922328613750390, next:4960000 invalid dropping st:0
      DTS -922328613750387, next:4990000 st:0 invalid dropping
      PTS -922328613750387, next:4990000 invalid dropping st:0
      DTS -922328613750384, next:5020000 st:0 invalid dropping
      PTS -922328613750384, next:5020000 invalid dropping st:0
      DTS -922328613750381, next:5050000 st:0 invalid dropping
      PTS -922328613750381, next:5050000 invalid dropping st:0
      DTS -922328613750378, next:5080000 st:0 invalid dropping
      PTS -922328613750378, next:5080000 invalid dropping st:0
      DTS -922328613750375, next:5110000 st:0 invalid dropping
      PTS -922328613750375, next:5110000 invalid dropping st:0
      DTS -922328613750372, next:5140000 st:0 invalid dropping
      PTS -922328613750372, next:5140000 invalid dropping st:0
      DTS -922328613750369, next:5170000 st:0 invalid dropping
      PTS -922328613750369, next:5170000 invalid dropping st:0
      DTS -922328613750366, next:5200000 st:0 invalid dropping
      PTS -922328613750366, next:5200000 invalid dropping st:0
      DTS -922328613750363, next:5230000 st:0 invalid dropping
      PTS -922328613750363, next:5230000 invalid dropping st:0
      DTS -922328613750360, next:5260000 st:0 invalid dropping
      PTS -922328613750360, next:5260000 invalid dropping st:0
      DTS -922328613750357, next:5290000 st:0 invalid dropping
      PTS -922328613750357, next:5290000 invalid dropping st:0
      DTS -922328613750354, next:5320000 st:0 invalid dropping
      PTS -922328613750354, next:5320000 invalid dropping st:0
      DTS -922328613750351, next:5350000 st:0 invalid dropping
      PTS -922328613750351, next:5350000 invalid dropping st:0
      DTS -922328613750348, next:5380000 st:0 invalid dropping
      PTS -922328613750348, next:5380000 invalid dropping st:0
      DTS -922328613750345, next:5410000 st:0 invalid dropping
      PTS -922328613750345, next:5410000 invalid dropping st:0
      DTS -922328613750342, next:5440000 st:0 invalid dropping
      PTS -922328613750342, next:5440000 invalid dropping st:0
      DTS -922328613750339, next:5470000 st:0 invalid dropping
      PTS -922328613750339, next:5470000 invalid dropping st:0
      DTS -922328613750336, next:5500000 st:0 invalid dropping
      PTS -922328613750336, next:5500000 invalid dropping st:0
      DTS -922328613750333, next:5530000 st:0 invalid dropping
      PTS -922328613750333, next:5530000 invalid dropping st:0
      DTS -922328613750330, next:5560000 st:0 invalid dropping
      PTS -922328613750330, next:5560000 invalid dropping st:0
      frame=  223 fps= 55 q=0.0 q=31.0 size=     105kB time=00:00:05.58 bitrate= 154.1kbits/s dup=56 drop=0    
      DTS -922328613750327, next:5590000 st:0 invalid dropping
      PTS -922328613750327, next:5590000 invalid dropping st:0
      DTS -922328613750324, next:5620000 st:0 invalid dropping
      PTS -922328613750324, next:5620000 invalid dropping st:0
      DTS -922328613750321, next:5650000 st:0 invalid dropping
      PTS -922328613750321, next:5650000 invalid dropping st:0
      DTS -922328613750312, next:5740000 st:0 invalid dropping
      PTS -922328613750312, next:5740000 invalid dropping st:0
      DTS -922328613750303, next:5830000 st:0 invalid dropping
      PTS -922328613750303, next:5830000 invalid dropping st:0
      DTS -922328613750294, next:5920000 st:0 invalid dropping
      PTS -922328613750294, next:5920000 invalid dropping st:0
      DTS -922328613750285, next:6010000 st:0 invalid dropping
      PTS -922328613750285, next:6010000 invalid dropping st:0
      DTS -922328613750276, next:6100000 st:0 invalid dropping
      PTS -922328613750276, next:6100000 invalid dropping st:0
      DTS -922328613750267, next:6190000 st:0 invalid dropping
      PTS -922328613750267, next:6190000 invalid dropping st:0
      DTS -922328613750258, next:6280000 st:0 invalid dropping
      PTS -922328613750258, next:6280000 invalid dropping st:0
      DTS -922328613750249, next:6370000 st:0 invalid dropping
      PTS -922328613750249, next:6370000 invalid dropping st:0
      DTS -922328613750240, next:6460000 st:0 invalid dropping
      PTS -922328613750240, next:6460000 invalid dropping st:0
      DTS -922328613750231, next:6550000 st:0 invalid dropping
      PTS -922328613750231, next:6550000 invalid dropping st:0
      DTS -922328613750222, next:6640000 st:0 invalid dropping
      PTS -922328613750222, next:6640000 invalid dropping st:0
      DTS -922328613750213, next:6730000 st:0 invalid dropping
      PTS -922328613750213, next:6730000 invalid dropping st:0
      frame=  238 fps= 52 q=0.0 q=31.0 size=     355kB time=00:00:06.76 bitrate= 429.7kbits/s dup=100 drop=0    
      DTS -922328613750204, next:6820000 st:0 invalid dropping
      PTS -922328613750204, next:6820000 invalid dropping st:0
      DTS -922328613750195, next:6910000 st:0 invalid dropping
      PTS -922328613750195, next:6910000 invalid dropping st:0
      DTS -922328613750186, next:7000000 st:0 invalid dropping
      PTS -922328613750186, next:7000000 invalid dropping st:0
      DTS -922328613750177, next:7090000 st:0 invalid dropping
      PTS -922328613750177, next:7090000 invalid dropping st:0
      DTS -922328613750168, next:7180000 st:0 invalid dropping
      PTS -922328613750168, next:7180000 invalid dropping st:0
      DTS -922328613750159, next:7270000 st:0 invalid dropping
      PTS -922328613750159, next:7270000 invalid dropping st:0
      DTS -922328613750150, next:7360000 st:0 invalid dropping
      PTS -922328613750150, next:7360000 invalid dropping st:0
      DTS -922328613750141, next:7450000 st:0 invalid dropping
      PTS -922328613750141, next:7450000 invalid dropping st:0
      DTS -922328613750132, next:7540000 st:0 invalid dropping
      PTS -922328613750132, next:7540000 invalid dropping st:0
      DTS -922328613750123, next:7630000 st:0 invalid dropping
      PTS -922328613750123, next:7630000 invalid dropping st:0
      DTS -922328613750114, next:7720000 st:0 invalid dropping
      PTS -922328613750114, next:7720000 invalid dropping st:0
      DTS -922328613750105, next:7810000 st:0 invalid dropping
      PTS -922328613750105, next:7810000 invalid dropping st:0
      DTS -922328613750096, next:7900000 st:0 invalid dropping
      PTS -922328613750096, next:7900000 invalid dropping st:0
      frame=  251 fps= 50 q=0.0 q=31.0 size=     424kB time=00:00:07.92 bitrate= 438.9kbits/s dup=145 drop=0    
      DTS -922328613750087, next:7990000 st:0 invalid dropping
      PTS -922328613750087, next:7990000 invalid dropping st:0
      DTS -922328613750078, next:8080000 st:0 invalid dropping
      PTS -922328613750078, next:8080000 invalid dropping st:0
      DTS -922328613750069, next:8170000 st:0 invalid dropping
      PTS -922328613750069, next:8170000 invalid dropping st:0
      DTS -922328613750060, next:8260000 st:0 invalid dropping
      PTS -922328613750060, next:8260000 invalid dropping st:0
      DTS -922328613750051, next:8350000 st:0 invalid dropping
      PTS -922328613750051, next:8350000 invalid dropping st:0
      DTS -922328613750042, next:8440000 st:0 invalid dropping
      PTS -922328613750042, next:8440000 invalid dropping st:0
      DTS -922328613750033, next:8530000 st:0 invalid dropping
      PTS -922328613750033, next:8530000 invalid dropping st:0
      DTS -922328613750024, next:8620000 st:0 invalid dropping
      PTS -922328613750024, next:8620000 invalid dropping st:0
      DTS -922328613750015, next:8710000 st:0 invalid dropping
      PTS -922328613750015, next:8710000 invalid dropping st:0
      DTS -922328613750006, next:8800000 st:0 invalid dropping
      PTS -922328613750006, next:8800000 invalid dropping st:0
      DTS -922328613749997, next:8890000 st:0 invalid dropping
      PTS -922328613749997, next:8890000 invalid dropping st:0
      DTS -922328613749988, next:8980000 st:0 invalid dropping
      PTS -922328613749988, next:8980000 invalid dropping st:0
      DTS -922328613749979, next:9070000 st:0 invalid dropping
      PTS -922328613749979, next:9070000 invalid dropping st:0
      DTS -922328613749970, next:9160000 st:0 invalid dropping
      PTS -922328613749970, next:9160000 invalid dropping st:0
      DTS -922328613749961, next:9250000 st:0 invalid dropping
      PTS -922328613749961, next:9250000 invalid dropping st:0
      frame=  266 fps= 48 q=0.0 q=31.0 size=     514kB time=00:00:09.28 bitrate= 454.1kbits/s dup=198 drop=0    
      DTS -922328613749952, next:9340000 st:0 invalid dropping
      PTS -922328613749952, next:9340000 invalid dropping st:0
      DTS -922328613749943, next:9430000 st:0 invalid dropping
      PTS -922328613749943, next:9430000 invalid dropping st:0
      DTS -922328613749934, next:9520000 st:0 invalid dropping
      PTS -922328613749934, next:9520000 invalid dropping st:0
      DTS -922328613749925, next:9610000 st:0 invalid dropping
      PTS -922328613749925, next:9610000 invalid dropping st:0
      DTS -922328613749916, next:9700000 st:0 invalid dropping
      PTS -922328613749916, next:9700000 invalid dropping st:0
      DTS -922328613749907, next:9790000 st:0 invalid dropping
      PTS -922328613749907, next:9790000 invalid dropping st:0
      DTS -922328613749898, next:9880000 st:0 invalid dropping
      PTS -922328613749898, next:9880000 invalid dropping st:0
      DTS -922328613749889, next:9970000 st:0 invalid dropping
      PTS -922328613749889, next:9970000 invalid dropping st:0
      DTS -922328613749880, next:10060000 st:0 invalid dropping
      PTS -922328613749880, next:10060000 invalid dropping st:0
      frame=  274 fps= 47 q=0.0 Lq=-1.0 size=     522kB time=00:00:10.00 bitrate= 427.6kbits/s dup=226 drop=0    
    
      video:1123kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
      [libx264 @ 0000000000388780] frame I:10    Avg QP:22.68  size: 39473
      [libx264 @ 0000000000388780] frame P:213   Avg QP:18.37  size:   932
      [libx264 @ 0000000000388780] frame B:277   Avg QP:22.48  size:    99
      [libx264 @ 0000000000388780] consecutive B-frames: 24.2%  3.2%  7.8% 64.8%
      [libx264 @ 0000000000388780] mb I  I16..4: 18.8% 71.8%  9.4%
      [libx264 @ 0000000000388780] mb P  I16..4:  2.0%  1.5%  1.1%  P16..4:  2.5%  0.7%  0.3%  0.0%  0.0%    skip:92.0%
      [libx264 @ 0000000000388780] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  2.8%  0.1%  0.0%  direct: 0.1%  skip:97.0%  L0:53.3% L1:45.4% BI: 1.3%
      [libx264 @ 0000000000388780] 8x8 transform intra:52.6% inter:18.8%
      [libx264 @ 0000000000388780] coded y,u,v intra: 29.3% 16.5% 16.1% inter: 0.3% 0.1% 0.1%
      [libx264 @ 0000000000388780] i16 v,h,dc,p: 53% 19% 28%  0%
      [libx264 @ 0000000000388780] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 22%  7% 61%  4%  4%  0%  1%  0%  1%
      [libx264 @ 0000000000388780] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 38% 20% 28%  5%  6%  1%  1%  0%  1%
      [libx264 @ 0000000000388780] Weighted P-Frames: Y:1.4% UV:0.5%
      [libx264 @ 0000000000388780] ref P L0: 66.8% 10.5% 19.4%  3.1%  0.1%
      [libx264 @ 0000000000388780] ref B L0: 59.5% 31.7%  8.8%
      [libx264 @ 0000000000388780] ref B L1: 96.9%  3.1%
      [libx264 @ 0000000000388780] kb/s:496.63
    

  • sockdevs

    @apapadimoulis said:

    I logged on to the server and it said out of memory again.So, then I shut it down and upgraded to 4GB.

    I blame Ruby for having such poor rubbish collection, and Linux for being generally rubbish.

    What's really funny is that I run a bunch of forums on a smaller Linux VPS than this one currently runs on.

    The fact Discourse requires 1GB of RAM just to RUN (I ran my first forum on a 160MB RAM VPS), and 2GB to be usable is not a fault of Linux. Might be Ruby's but there are some serious architectural issues.

    I also look after forums larger than this one on smaller hardware. Linux ain't the problem.



  • Discourse needs 2GB of RAM because it has to have its own installation of Linux with all the database programs and other stuff installed. It's definitely an architecture problem. GitHub is written in Ruby on Rails and it doesn't go down every three hours due to shitting its pants.

    <It goes down every three hours due to hardware failing!>


  • sockdevs

    Yes... but I had my own 160MB RAM VPS running its own instance of the Linux kernel and everything. FFS I initially tried running Red Hat with X on that VPS to start with and it actually worked better than expected (this was in 2006)

    I have a VPS. It runs MySQL, Apache, PHP and a ton of other crap and there's simply no problems with this stuff.



  • Seems like it should be pretty simple to figure out what was eating so much memory. Obviously, a DB can be scaled up a lot to keep things in memory, but presumably that sort of thing is fixed and shouldn't grow over time (psql gets this right, doesn't it?).

    Plus they're using redis to cache stuff, right? That seems like an obvious place, but it should be more obvious to someone with shell access to be able to figure out.



  • I can run an instance on sitecore on less ram than discourse needs and it is a very large .net cms



  • A 1-hour, 50fps gif is smaller than Discourse's RAM usage.

    Edit: Proof:



  • We might as well have a SharePoint forum


  • Grade A Premium Asshole

    @VaelynPhi said:

    I used to have to use Redmine for issue tracking and the damned thing takes as much as 30 seconds just to show the login screen sometimes.

    You had some config issues with your environment. I have done dev work with Redmine and it is pretty responsive even just running on the built-in rails server, sqlite and running on a VM in WMWare Workstation with only 2GB of RAM and the full Gnome desktop environment loaded. Page loads there were ~100ms. Running on a production environment, in a VM with 1GB RAM and database on a separate server, page loads are in low double digit milliseconds.

    I have seen roughly the same out of the forks of Redmine with extended functionality.

    @ben_lubar said:

    Discourse needs 2GB of RAM because it has to have its own installation of Linux with all the database programs and other stuff installed.

    So, their backups each run individually on each instance? Why not tie them all to one db server and then just slave that and run snapshots and backups there? Jesus fuck, that is wasteful of resources and kind of kills the point of using Docker (or any containerized virtualization). How are db backups setup anyway?

    @Arantor said:

    The fact Discourse requires 1GB of RAM just to RUN (I ran my first forum on a 160MB RAM VPS), and 2GB to be usable is not a fault of Linux. Might be Ruby's but there are some serious architectural issues.

    Definitely not a problem of Ruby. They have seriously fixed some interpreter issues in recent releases. I just started evaluating it again and I am impressed with all of the improvements. Ruby > 1.8 (IIRC) has major performance improvements and reduced overhead.


  • Discourse touched me in a no-no place

    @powerlord said:

    You had me up until there.

    Yes, but @blakeyrat's all kinds of happy.


  • Grade A Premium Asshole

    @lucas said:

    We might as well have a SharePoint forum

    The evil ideas thread is....somewhere else.



  • @Intercourse said:

    How are db backups setup anyway?

    They have an option to use Amazon S3:

    I have no idea if that's how we're set up or what.



  • @boomzilla said:

    Seems like it should be pretty simple to figure out what was eating so much memory.

    Yep. Discourse. It's got pathologically bad UI design, albeit with a few (mostly flawed implementations of) good ideas, it causes browser crashes by just scrolling and eats network bandwidth like a fat bastard scoffs pies; what makes you think it's somehow not equally awful at the server end?

    The other options would be Rails, redis or PostgreSQL.

    RoR requires a certain amount of memory, but none of my live rails-based sites and systems have ever gone this far AWOL, and they were all originally implemented using ruby < 1.8.4 on machines with less than 2GB of memory. One was a micropayment system with a proven capability of thousands of transactions a second...

    redis and (above all) postgres are good.



  • @tufty said:

    what makes you think it's somehow not equally awful at the server end?

    I don't. I'm just saying it shouldn't be such a mystery.



  • @VaelynPhi said:

    Given that I've run shitty-as-all-get-out Wordpress on a linux box with 2GB, I think we can place the blame squarely on Ruby.

    I wouldn't blame Ruby. I would blame whoever wrote the problematic code.

    Seriously, any language with a garbage collector is only as good as the people writing code for it in terms of memory consumption.

    To be able to garbage collect objects, they have to be abandoned. This might sound as a trifle, but if you don't understand the concept of garbage collection, or the paradigm of the gc language you use, it's not exactly difficult to create situations in which objects live far longer than necessary.

    And you might say: "Ok, so the object gets deleted a bit later, what's the deal?" The deal is that if objects whose memory footprint is 1GB get deleted long after it would be possible with better programming, do need those 1GB of RAM nevertheless.

    This is 2014. Memory is cheap, so people have a tendency to approach memory issues with "simply buy more RAM" instead of coding defensively in that respect.

    But if your server is a cloud server getting more RAM for you means buying a bigger plan, and by doing so forking out more money per month, I'd call that attitude extremely customer-unfriendly.



  • @boomzilla said:

    I'm just saying it shouldn't be such a mystery.

    I was trying to say that it's not.



  • @Intercourse said:

    They have seriously fixed some interpreter issues in recent releases. I just started evaluating it again and I am impressed with all of the improvements. Ruby > 1.8 (IIRC) has major performance improvements and reduced overhead.

    This.

    The major issue with the 1.8 runtime was its approach to dynamic code - it assumed that as everything could be dynamic, that everything should be treated as though it were dynamic. The result was an inability to cache anything internally, re-evaluating the syntax every time a line of code was interpreted. This is a large part of why it was possible to write an interpreter for most of Ruby in interpreted Lisp, and have it be faster than Ruby. It's also why I have a local fork of the 1.8.4 ruby codebase which doesn't handle eval in normal code.

    1.9 is a load better. Between 2 and 5 times faster than "vanilla" 1.8.x, with 2/3 the memory usage, according to my benchmarks.


  • Grade A Premium Asshole

    But that means that all of the containerized instances of Discourse running over at DiscoHQ all run distinct backup tasks? That seems wasteful when you could just tie them all back to a single db instance and back it all up at once, slave it for failover, etc.



  • You were using 2 GB of that really, really cheap stuff?



  • I have no clue how they're set up. But not everyone has the discodevs doing the hosting. My understanding is that their idea is that you get a vm at DigitalOcean and put docker on there and you get all the stuff (pgsql, redis, ror, all their code, etc) and off you go.


  • Grade A Premium Asshole

    OK, that makes sense then. I was assuming that we were on an instance being resold by DiscoInc.


  • Discourse touched me in a no-no place

    @tufty said:

    1.9 is a load better. Between 2 and 5 times faster than "vanilla" 1.8.x, with 2/3 the memory usage, according to my benchmarks.

    If I remember 1.8 right, being 2–5 times faster than it is not quite as big an achievement as one might hope. (I'm experimenting with some code that does clever things to get a 66 times speedup1. But I'm not using Ruby, so I'm starting from a faster baseline.)

    1 Very experimental testing though, probably won't be as good with normal code. And the details of LLVM make my head ache…



  • Yeah.... @apapapapaidmolollsldl...uh...auto-complete isn't working to summon Alex...he could confirm, but I don't think we're on one of their plans, but we have the friends and family plan for support, which I think means that in addition to local DC admin accounts, they have server access to @apapadimoulis's DigitalOcean server.

    That's the story that my shoulder aliens have put together, at any rate.


  • Grade A Premium Asshole

    The real question is, who fellates who in that arrangement?



  • @Intercourse said:

    VaelynPhi:
    I used to have to use Redmine for issue tracking and the damned thing takes as much as 30 seconds just to show the login screen sometimes.

    You had some config issues with your environment. I have done dev work with Redmine and it is pretty responsive even just running on the built-in rails server, sqlite and running on a VM in WMWare Workstation with only 2GB of RAM and the full Gnome desktop environment loaded. Page loads there were ~100ms. Running on a production environment, in a VM with 1GB RAM and database on a separate server, page loads are in low double digit milliseconds.

    I have seen roughly the same out of the forks of Redmine with extended functionality.

    This is certainly possible, but as the server was set up according to Redmine's own documentation, I would expect it to run decently. Exactly what kind of config issues would you expect? At some point we had a Rails enthusiast (who was supposedly also an expert) perform some "tune ups"; whatever he did did not seem to have much effect. Do note, BTW, that I was referring to the time to show the login screen; most page loads after login were acceptable (with some exceptions). As you might expect, it's the exceptions that were the most frustrating.


  • Grade A Premium Asshole

    Man, you just asked a million dollar question. What environment are you running: Ruby version, rails version, Redmine version, etc? Hardware it is running on?


  • Discourse touched me in a no-no place

    @Intercourse said:

    How are db backups setup anyway?

    Badly.

    @Matches - another bug for you.

    @boomzilla said:

    I have no idea if that's how we're set up or what.

    Given the above, I've no idea whether the S3 backups are complete.


  • Grade A Premium Asshole

    Wow. How is it possible to screw it up that badly? There are several good gems to do db backups. Is there something I am missing that makes DiscoBackups a special use-case?



  • @Intercourse said:

    Man, you just asked a million dollar question. What environment are you running: Ruby version, rails version, Redmine version, etc? Hardware it is running on?

    Alas, questions I can't really answer with much certainty anymore since I quit that job back in May. I can say it was the latest of all three I could in July of last year, and I may have updated them in December. I do remember it being a pain in the arse to use the actual latest versions, largely because coherent installation instructions weren't available.

    If I'm reading the history correctly, Redmine would have been 2.3, Ruby 2.1, and Rails I have no idea, but it was installed with Ruby's gem, so it would have been whatever version Ruby 2.1 made available in late December 2013. Also, it was December when I last updated the installtion. It was running on a machine with 64GB RAM, a quad-core i7 at 4+GHz/core, Ubuntu 12.04 (probably .2) on an SSD... IE, such that I'd expect the hardware specs to be almost irrelevant.


  • Discourse touched me in a no-no place

    @Intercourse said:

    Wow. How is it possible to screw it up that badly? There are several good gems to do db backups. Is there something I am missing that makes DiscoBackups a special use-case?

    The sizes listed on the page listing the backups are correct (in so far that they report larger sizes) - it's the downloads that are broken.


  • Grade A Premium Asshole

    @VaelynPhi said:

    I'd expect the hardware specs to be almost irrelevant.

    You have a talent for understatement.

    @VaelynPhi said:

    Alas, questions I can't really answer with much certainty anymore since I quit that job back in May.

    The point is moot then, but if I had to take a WAG as to what it could possibly be, someone was using it with the built-in webrick web server. It is great for dev usage, but it can only process one request at a time. Zero concurrency. That is a WAG though.


  • Grade A Premium Asshole

    So the local db backups are presumably fine, but they become FUBAR when downloaded or sent to S3?


  • Discourse touched me in a no-no place

    Downloaded yes. No idea about S3. Would have to ask @apapadimoulis about that.


  • Grade A Premium Asshole

    I sit here with a furled brow wondering how they could fuck that up. We have been able to transfer files over HTTP for a while now. I mean...that point of the internets is pretty well figured out. What size are the DLs?



  • If you looked at the linked thread, @sam commented that he suspected a misconfigured bit of nginx.


  • Discourse touched me in a no-no place

    @Intercourse said:

    What size are the DLs?

    Are you ready for this?

    [spoiler]

    [/spoiler]

    I appear to be misremembering the incrementing sizes given, and they appear to be discorounded - one of the above:

    [spoiler]
    [pjh@lenovo ~]$ ls /tmp/2014-09-19-030001.tar.gz -lh
    -rw-rw-r-- 1 pjh pjh 1.1G Sep 19 10:43 /tmp/2014-09-19-030001.tar.gz
    [/spoiler]
    (And that's using 2^30, not 10^9 which technically would be larger, but the output using --sis to get it results in the same display.)

    @boomzilla said:

    If you looked at the linked thread, @sam commented that he suspected a misconfigured bit of nginx.

    But not yet confirmed.


  • Banned

    @PJH if you need to download backups its probably safer for now for you do create a new backup without images and then download it (should be 100mb)

    Still trying to figure out why linux is allocating so much memory for large downloads.


  • Grade A Premium Asshole

    @PJH said:

    Are you ready for this?

    Holy shit, the entirety of Wikipedia is <20GB (sans images, of course). How is it possible for our forum to be that large?


  • Banned

    Images ... lots of images...



  • Seems like that got turned off.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.