@ben_lubar tries to parse <del>ServerCooties</del> data



  • @ben_lubar said in Refreshrefreshrefreshrefresh:

    @accalia said in Refreshrefreshrefreshrefresh:

    @sloosecannon said in Refreshrefreshrefreshrefresh:

    Paging @accalia - what does servercooties say about the update at 11:09:22 EST today?

    have fun.

    ben@australium:~$ mkdir servercooties
    ben@australium:~$ cd servercooties/
    ben@australium:~/servercooties$ ls
    ben@australium:~/servercooties$ wget https://servercooties.io/static/servercooties.sql.zip
    --2016-07-01 21:13:18--  https://servercooties.io/static/servercooties.sql.zip
    Resolving servercooties.io (servercooties.io)... 45.55.146.244, 2604:a880:800:10::76a:f001
    Connecting to servercooties.io (servercooties.io)|45.55.146.244|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 69915409 (67M) [application/zip]
    Saving to: ‘servercooties.sql.zip’
    
    servercooties.sql.z 100%[===================>]  66.68M  1018KB/s    in 53s     
    
    2016-07-01 21:14:11 (1.26 MB/s) - ‘servercooties.sql.zip’ saved [69915409/69915409]
    
    ben@australium:~/servercooties$ unzip servercooties.sql.zip 
    Archive:  servercooties.sql.zip
      inflating: var/www/SockSite/servercooties.sql  
    ben@australium:~/servercooties$ sqlite3 var/www/SockSite/servercooties.sql
    SQLite version 3.11.0 2016-02-15 17:29:24
    Enter ".help" for usage hints.
    sqlite> .tables
    checks2  pages  
    sqlite> select count(*) from checks2;
    Error: database disk image is malformed
    

    When you say "fun", do you mean ben_lubot?

    Interesting idea from a StackOverflow user: http://stackoverflow.com/a/15874884/2664560

    ben@australium:~/servercooties$ cat <( sqlite3 var/www/SockSite/servercooties.sql .dump | grep "^ROLLBACK" -v ) <( echo "COMMIT;" ) | time sqlite3 fixed_servercooties.db
    Error: near line 7059611: cannot commit - no transaction is active
    Command exited with non-zero status 1
    93.82user 1.00system 1:35.82elapsed 98%CPU (0avgtext+0avgdata 5764maxresident)k
    0inputs+314712outputs (0major+806minor)pagefaults 0swaps
    ben@australium:~/servercooties$ sqlite3 fixed_servercooties.db 
    SQLite version 3.11.0 2016-02-15 17:29:24
    Enter ".help" for usage hints.
    sqlite> select count(*) from checks2;
    7059111
    sqlite> 
    


  • 0_1467437880283_upload-3579f4bf-51b5-45ba-94e3-0b058f887ada


  • Discourse touched me in a no-no place

    @ben_lubar What's on the vertical axis? (I see that events have been more frequent since mid-May, but not what the significance of that is.)


  • sockdevs

    @ben_lubar said in @ben_lubar tries to parse ServerCooties data:

    Interesting idea from a StackOverflow user: http://stackoverflow.com/a/15874884/2664560

    ooooh. that's what caused the corrupt table.....

    brb. automating that cleanup in the dump so you don't have to do it manually.



  • @dkf said in @ben_lubar tries to parse ServerCooties data:

    What's on the vertical axis?

    Response time of the front page of the forum.



  • @ben_lubar just so I too can experience the joy of telling someone how to do a job they already know how to do, is there any recurring daily pattern?



  • I seem to have accidentally invented Discourse

    0_1467582383016_upload-e59f64a2-e071-4b96-b7b0-b019a50dceb4




  • sockdevs

    @ben_lubar said in @ben_lubar tries to parse ServerCooties data:

    Error: database disk image is malformed

    accalia@isitjustmeorservercooties:~$ cat makebackup.sh 
    #!/bin/bash
    
    TEMP="$(mktemp -d)"
    cd  "${TEMP}"
    sqlite3 /var/www/SockSite/servercooties.sql '.dump' | sqlite3 servercooties.sql
    zip servercooties.sql.zip servercooties.sql
    mv servercooties.sql.zip /var/www/SockSite/static/servercooties.sql.zip
    cd ..
    rm -r "${TEMP}"
    accalia@isitjustmeorservercooties:~$ crontab -l
    0 0 * * * nice /home/accalia/makebackup.sh
    accalia@isitjustmeorservercooties:~$ 
    

    nevar again should you have that issue.

    of course you may need to tweak your script if you pull the zip file and process automatically as the superfluous path components in the zip file have been removed.



  • @accalia said in @ben_lubar tries to parse ServerCooties data:

    you may need to tweak your script

    My script looks for any file in the zip, and if there are multiple files or no files, it errors.



  • Excerpt from the output of my NodeBB !!SCIENCE!!-ifier:

    2016/07/05 13:02:48 user:1:"ben_lubar"
    2016/07/05 13:02:48 user:2:"zogstrip"
    2016/07/05 13:02:48 user:3:"Remy"
    2016/07/05 13:02:48 user:4:"apapadimoulis"
    2016/07/05 13:02:48 user:5:"mark_bowytz"
    2016/07/05 13:02:48 user:6:"awinter"
    2016/07/05 13:02:48 user:7:"InedoJohn"
    2016/07/05 13:02:48 user:8:"kharnagy"
    2016/07/05 13:02:48 user:9:"PaulaBean"
    2016/07/05 13:02:48 user:10:"prashantP"
    2016/07/05 13:02:48 user:11:"morbiuswilters"
    2016/07/05 13:02:48 user:12:"dhromed"
    2016/07/05 13:02:48 user:13:"blakeyrat"
    2016/07/05 13:02:51 user:14:"boomzilla"
    2016/07/05 13:02:51 user:15:"moderator"
    2016/07/05 13:02:56 user:16:"PJH"
    2016/07/05 13:02:56 user:17:"ammoQ"
    2016/07/05 13:02:56 user:18:"DaveK"
    2016/07/05 13:02:56 user:19:"El_Heffe"
    2016/07/05 13:02:56 user:20:"Cassidy"
    2016/07/05 13:03:00 user:21:"Lorne Kates"
    2016/07/05 13:03:01 user:22:"snoofle"
    2016/07/05 13:03:02 user:23:"error"
    2016/07/05 13:03:02 user:24:"frits"
    2016/07/05 13:03:07 user:25:"Zecc"
    2016/07/05 13:03:08 user:26:"asuffield"
    2016/07/05 13:03:08 user:27:"derula"
    2016/07/05 13:03:23 user:28:"dkf"
    2016/07/05 13:03:23 user:29:"da_Doctah"
    2016/07/05 13:03:23 user:30:"serguey123"
    2016/07/05 13:03:23 user:31:"tster"
    2016/07/05 13:03:23 user:32:"Xyro"
    2016/07/05 13:03:23 user:33:"Sutherlands"
    2016/07/05 13:03:23 user:34:"C_Octothorpe"
    2016/07/05 13:03:23 user:35:"Ronald"
    2016/07/05 13:03:23 user:36:"Lingerance"
    2016/07/05 13:03:24 user:37:"Zemm"
    2016/07/05 13:03:28 user:38:"chubertdev"
    2016/07/05 13:03:31 user:39:"Weng"
    2016/07/05 13:03:31 user:40:"boog"
    2016/07/05 13:03:31 user:41:"hoodaticus"
    2016/07/05 13:03:31 user:42:"KattMan"
    2016/07/05 13:03:31 user:43:"Matt_Westwood"
    2016/07/05 13:03:31 user:44:"PSWorx"
    2016/07/05 13:03:33 user:45:"flabdablet"
    2016/07/05 13:03:33 user:46:"SpectateSwamp"
    2016/07/05 13:03:33 user:47:"DOA"
    2016/07/05 13:03:33 user:48:"real_aardvark"
    2016/07/05 13:03:33 user:49:"Spectre"
    2016/07/05 13:03:33 user:50:"MiffTheFox"
    2016/07/05 13:03:33 user:51:"Zylon"
    2016/07/05 13:03:33 user:52:"AbbydonKrafts"
    2016/07/05 13:03:34 user:53:"Code_Dependent"
    2016/07/05 13:03:34 user:54:"operagost"
    2016/07/05 13:03:34 user:55:"Jaime"
    2016/07/05 13:03:34 user:56:"Jeff_S"
    2016/07/05 13:03:34 user:57:"TheCPUWizard"
    2016/07/05 13:03:34 user:58:"ObiWayneKenobi"
    2016/07/05 13:03:34 user:59:"OzPeter"
    2016/07/05 13:03:34 user:60:"KenW"
    2016/07/05 13:03:34 user:61:"danixdefcon5"
    2016/07/05 13:03:34 user:62:"FredSaw"
    2016/07/05 13:03:34 user:63:"brazzy"
    2016/07/05 13:03:52 user:64:"mott555"
    2016/07/05 13:03:52 user:65:"dtech"
    2016/07/05 13:03:52 user:66:"TGV"
    2016/07/05 13:03:52 user:67:"Mason_Wheeler"
    2016/07/05 13:03:52 user:68:"galgorah"
    2016/07/05 13:03:52 user:69:"lolwtf"
    2016/07/05 13:03:53 user:70:"Carnildo"
    2016/07/05 13:03:53 user:71:"tgape"
    2016/07/05 13:03:53 user:72:"Severity_One"
    2016/07/05 13:03:53 user:73:"bjolling"
    2016/07/05 13:03:53 user:74:"pjt33"
    2016/07/05 13:03:53 user:75:"CodeSimian"
    2016/07/05 13:03:53 user:76:"vt_mruhlin"
    2016/07/05 13:03:54 user:77:"amischiefr"
    2016/07/05 13:03:59 user:78:"locallunatic"
    2016/07/05 13:04:00 user:79:"Scarlet_Manuka"
    2016/07/05 13:04:00 user:80:"ParkinT"
    2016/07/05 13:04:01 user:81:"mikeTheLiar"
    2016/07/05 13:04:01 user:82:"The_Quiet_One"
    2016/07/05 13:04:01 user:83:"eViLegion"
    2016/07/05 13:04:01 user:84:"Random832"
    2016/07/05 13:04:01 user:85:"stratos"
    2016/07/05 13:04:01 user:86:"RayS"
    2016/07/05 13:04:01 user:87:"Steve_The_Cynic"
    2016/07/05 13:04:01 user:88:"bridget99"
    2016/07/05 13:04:01 user:89:"Maurits"
    2016/07/05 13:04:02 user:90:"dlikhten"
    2016/07/05 13:04:02 user:91:"shadowman"
    2016/07/05 13:04:02 user:92:"dubwai"
    2016/07/05 13:04:02 user:93:"emurphy"
    2016/07/05 13:04:02 user:94:"m0ffx"
    2016/07/05 13:04:02 user:95:"Coyne"
    2016/07/05 13:04:02 user:96:"Welbog"
    2016/07/05 13:04:02 user:97:"savar"
    2016/07/05 13:04:02 user:98:"Eternal_Density"
    2016/07/05 13:04:02 user:99:"ekolis"
    2016/07/05 13:04:03 user:100:"djork"
    

    At the time of posting, it's up to 2016/07/05 13:08:25 user:624:"Groaner"2016/07/05 13:13:16 user:1661:"varun"

    All it's doing is reading user data from the MongoDB database. It's not writing the data anywhere or trying to keep it all in memory.

    I had to add a call to SetCursorTimeout(0) because on the first test run, the cursor for the "get list of user IDs" query expired while I was iterating through it.

    We only have what, 140k users? This shouldn't take more than a few lifetimes.



  • 2016/07/05 13:17:41 user:95923:"Gawgrerge"
    2016/07/05 13:17:41 Cursor not found, cursor id: 28781399268
    

    :fu: MongoDB



  • By the way, here's my current code: https://play.golang.org/p/6wAZlwPoWF


  • sockdevs

    @ben_lubar said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    @accalia said in @ben_lubar tries to parse ServerCooties data:

    you may need to tweak your script

    My script looks for any file in the zip, and if there are multiple files or no files, it errors.

    you should be good then.







  • 0_1467750387730_upload-287836d2-35fc-4a68-8dad-f4d27024e271

    Go home htop you're drunk



  • So apparently the bug that's affecting me might be one that was fixed in the latest version. Time to update mongodb...





  • db.objects.find({_key: 'users:joindate'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).forEach(function(uid) { db.objects.findOne({_key: 'user:' + uid.value}); db.objects.findOne({_key: 'user:' + uid.value + ':settings'}); db.objects.find({_key: 'uid:' + uid.value + ':ip'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).forEach(function() { }); db.objects.find({_key: 'uid:' + uid.value + ':ignored:cids'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).forEach(function() { }); db.objects.find({_key: 'uid:' + uid.value + ':ignored_tids'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).forEach(function() { }); db.objects.find({_key: 'uid:' + uid.value + ':followed_tids'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).forEach(function() { }); db.objects.find({_key: 'uid:' + uid.value + ':favourites'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).forEach(function() { }); db.objects.find({_key: 'following:' + uid.value}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).forEach(function() { }); db.objects.find({_key: 'uid:' + uid.value + ':upvote'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).forEach(function() { }); db.objects.find({_key: 'uid:' + uid.value + ':downvote'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).forEach(function() { }); })
    

    Ok, either this has the same error as the Go version or I'm confused as shit.



  • The Go version is sending

            "query" : {
                    "find" : "objects",
                    "filter" : {
                            "_key" : "users:joindate"
                    },
                    "sort" : {
                            "score" : 1,
                            "value" : 1
                    },
                    "projection" : {
                            "score" : 1,
                            "_id" : 0,
                            "value" : 1
                    },
                    "skip" : 0
            },
    

    whereas the mongo shell is sending

            "query" : {
                    "find" : "objects",
                    "filter" : {
                            "_key" : "users:joindate"
                    },
                    "sort" : {
                            "score" : 1,
                            "value" : 1
                    },
                    "projection" : {
                            "_id" : 0,
                            "value" : 1,
                            "score" : 1
                    },
                    "noCursorTimeout" : true
            },
    

    It looks like the field is set here: https://github.com/mongodb/mongo/blob/r3.2.7/src/mongo/shell/query.js#L225-L227

    I think mongod is looking at the bson property instead of the wire format flag.


  • Banned

    @ben_lubar I had this cursor not found problem, and worked around with smaller queries.





  • 0_1467768625638_upload-11ef272c-0091-42e4-b42c-d8b9afa591e7

    Not having to wait 48 hours to click a button is obviously :doing_it_wrong:



  • @ben_lubar If you just used something other than Go you'd never have had this problem.



  • @boomzilla said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    @ben_lubar If you just used something other than Go you'd never have had this problem.

    If NodeBB used something other than MongoDB, I would have never had this problem.



  • postgres=# begin; 
    BEGIN
    postgres=# explain select post_id, count(*) filter (where up) as upvotes, count(*) filter (where not up) as downvotes, (select array_agg(c.user_id) from (select b.user_id from user_post_votes b where b.post_id = a.post_id and b.up order by b.last_changed desc limit 10) c) as recent_upvotes from user_post_votes a group by post_id order by upvotes desc limit 10;
                                                                    QUERY PLAN                                                                
    ------------------------------------------------------------------------------------------------------------------------------------------
     Limit  (cost=19125014779.85..19125014779.88 rows=10 width=9)
       ->  Sort  (cost=19125014779.85..19125015107.92 rows=131226 width=9)
             Sort Key: (count(*) FILTER (WHERE a.up)) DESC
             ->  GroupAggregate  (cost=10000367436.85..19125011944.11 rows=131226 width=9)
                   Group Key: a.post_id
                   ->  Sort  (cost=10000367436.85..10000373951.23 rows=2605751 width=9)
                         Sort Key: a.post_id
                         ->  Seq Scan on user_post_votes a  (cost=10000000000.00..10000045217.51 rows=2605751 width=9)
                   SubPlan 1
                     ->  Aggregate  (cost=69533.60..69533.61 rows=1 width=8)
                           ->  Limit  (cost=69533.44..69533.47 rows=10 width=16)
                                 ->  Sort  (cost=69533.44..69533.49 rows=20 width=16)
                                       Sort Key: b.last_changed DESC
                                       ->  Index Scan using user_post_votes_pkey on user_post_votes b  (cost=0.43..69533.01 rows=20 width=16)
                                             Index Cond: (post_id = a.post_id)
                                             Filter: up
    (16 rows)
    
    postgres=# create index user_post_votes_by_post on user_post_votes (post_id);
    CREATE INDEX
    postgres=# explain select post_id, count(*) filter (where up) as upvotes, count(*) filter (where not up) as downvotes, (select array_agg(c.user_id) from (select b.user_id from user_post_votes b where b.post_id = a.post_id and b.up order by b.last_changed desc limit 10) c) as recent_upvotes from user_post_votes a group by post_id order by upvotes desc limit 10;
                                                              QUERY PLAN                                                          
    ------------------------------------------------------------------------------------------------------------------------------
     Limit  (cost=11123914.76..11123914.78 rows=10 width=9)
       ->  Sort  (cost=11123914.76..11124242.82 rows=131226 width=9)
             Sort Key: (count(*) FILTER (WHERE a.up)) DESC
             ->  GroupAggregate  (cost=0.43..11121079.01 rows=131226 width=9)
                   Group Key: a.post_id
                   ->  Index Scan using user_post_votes_by_post on user_post_votes a  (cost=0.43..144009.93 rows=2605751 width=9)
                   SubPlan 1
                     ->  Aggregate  (cost=83.48..83.49 rows=1 width=8)
                           ->  Limit  (cost=83.33..83.35 rows=10 width=16)
                                 ->  Sort  (cost=83.33..83.38 rows=20 width=16)
                                       Sort Key: b.last_changed DESC
                                       ->  Bitmap Heap Scan on user_post_votes b  (cost=4.58..82.90 rows=20 width=16)
                                             Recheck Cond: (post_id = a.post_id)
                                             Filter: up
                                             ->  Bitmap Index Scan on user_post_votes_by_post  (cost=0.00..4.58 rows=20 width=0)
                                                   Index Cond: (post_id = a.post_id)
    (16 rows)
    
    postgres=# create index user_post_votes_latest_upvotes_per_post on user_post_votes (post_id, last_changed desc) where up;
    CREATE INDEX
    postgres=# explain select post_id, count(*) filter (where up) as upvotes, count(*) filter (where not up) as downvotes, (select array_agg(c.user_id) from (select b.user_id from user_post_votes b where b.post_id = a.post_id and b.up order by b.last_changed desc limit 10) c) as recent_upvotes from user_post_votes a group by post_id order by upvotes desc limit 10;
                                                                         QUERY PLAN                                                                     
    ----------------------------------------------------------------------------------------------------------------------------------------------------
     Limit  (cost=5761560.03..5761560.06 rows=10 width=9)
       ->  Sort  (cost=5761560.03..5761888.10 rows=131226 width=9)
             Sort Key: (count(*) FILTER (WHERE a.up)) DESC
             ->  GroupAggregate  (cost=0.43..5758724.28 rows=131226 width=9)
                   Group Key: a.post_id
                   ->  Index Scan using user_post_votes_by_post on user_post_votes a  (cost=0.43..144009.93 rows=2605751 width=9)
                   SubPlan 1
                     ->  Aggregate  (cost=42.62..42.63 rows=1 width=8)
                           ->  Limit  (cost=0.43..42.49 rows=10 width=16)
                                 ->  Index Scan using user_post_votes_latest_upvotes_per_post on user_post_votes b  (cost=0.43..84.55 rows=20 width=16)
                                       Index Cond: (post_id = a.post_id)
    (11 rows)
    
    postgres=# rollback;
    

    Can I create an index on a GROUP BY?



  • @ben_lubar said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    @boomzilla said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    @ben_lubar If you just used something other than Go you'd never have had this problem.

    If NodeBB used something other than MongoDB, I would have never had this problem.

    If TDWTF used something other than NodeBB... :trolleybus:


  • Discourse touched me in a no-no place

    @ben_lubar said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    Can I create an index on a GROUP BY?

    How does that differ from creating an index on the thing that you're grouping by? (post_id?)



  • @ben_lubar said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    Can I create an index on a GROUP BY?

    Two responses come to mind:

    1. That's gibberish

    2. If you put the result in a temp table, you can put whatever indexes on the temp table you want, so knock yourself out

    And your pasting of that gibberish really makes me appreciate that SQL Server has a graphical tool to generate those reports in an actually-readable way.



  • 2016-07-07T22:39:17.423+0000 I COMMAND  [conn58] command 0.objects command: find { find: "objects", filter: { _key: "posts:pid" }, sort: { score: 1, value: 1 }, projection: { _id: 0, value: 1, score: 1 }, skip: 0, noCursorTimeout: true } keyUpdates:0 writeConflicts:0 numYields:44002 reslen:194 locks:{ Global: { acquireCount: { r: 88006 } }, MMAPV1Journal: { acquireCount: { r: 44007 } }, Database: { acquireCount: { r: 44003 } }, Collection: { acquireCount: { R: 44003 }, acquireWaitCount: { R: 4 }, timeAcquiringMicros: { R: 16888 } } } protocol:op_query 418920ms
    2016/07/07 17:41:26 errmsg: "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit."
    

    This is the equivalent of SELECT id, created_at FROM posts ORDER BY created_at, id taking 7 minutes to return an error about memory usage ON A TABLE WITH INDEXES ON BOTH OF THE FIELDS USED IN THE QUERY.



  • > db.objects.find({_key: 'posts:pid'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).explain()
    2016-07-07T22:48:10.243+0000 E QUERY    [thread1] Error: explain failed: {
            "ok" : 0,
            "errmsg" : "errmsg: \"Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.\"",
            "code" : 96
    } :
    _getErrorWithCode@src/mongo/shell/utils.js:25:13
    throwOrReturn@src/mongo/shell/explainable.js:31:1
    constructor/this.finish@src/mongo/shell/explain_query.js:176:24
    DBQuery.prototype.explain@src/mongo/shell/query.js:508:12
    @(shell):1:1
    

    :facepalm:



  • > db.objects.find({_key: 'posts:pid'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1}).explain()
    {
            "queryPlanner" : {
                    "plannerVersion" : 1,
                    "namespace" : "0.objects",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                            "_key" : {
                                    "$eq" : "posts:pid"
                            }
                    },
                    "winningPlan" : {
                            "stage" : "PROJECTION",
                            "transformBy" : {
                                    "_id" : 0,
                                    "value" : 1,
                                    "score" : 1
                            },
                            "inputStage" : {
                                    "stage" : "FETCH",
                                    "inputStage" : {
                                            "stage" : "IXSCAN",
                                            "keyPattern" : {
                                                    "_key" : 1,
                                                    "score" : -1
                                            },
                                            "indexName" : "_key_1_score_-1",
                                            "isMultiKey" : false,
                                            "isUnique" : false,
                                            "isSparse" : false,
                                            "isPartial" : false,
                                            "indexVersion" : 1,
                                            "direction" : "backward",
                                            "indexBounds" : {
                                                    "_key" : [
                                                            "[\"posts:pid\", \"posts:pid\"]"
                                                    ],
                                                    "score" : [
                                                            "[MinKey, MaxKey]"
                                                    ]
                                            }
                                    }
                            }
                    },
                    "rejectedPlans" : [
                            {
                                    "stage" : "PROJECTION",
                                    "transformBy" : {
                                            "_id" : 0,
                                            "value" : 1,
                                            "score" : 1
                                    },
                                    "inputStage" : {
                                            "stage" : "SORT",
                                            "sortPattern" : {
                                                    "score" : 1
                                            },
                                            "inputStage" : {
                                                    "stage" : "KEEP_MUTATIONS",
                                                    "inputStage" : {
                                                            "stage" : "SORT_KEY_GENERATOR",
                                                            "inputStage" : {
                                                                    "stage" : "FETCH",
                                                                    "inputStage" : {
                                                                            "stage" : "IXSCAN",
                                                                            "keyPattern" : {
                                                                                   "_key" : 1,
                                                                                   "value" : -1
                                                                            },
                                                                            "indexName" : "_key_1_value_-1",
                                                                            "isMultiKey" : false,
                                                                            "isUnique" : true,
                                                                            "isSparse" : true,
                                                                            "isPartial" : false,
                                                                            "indexVersion" : 1,
                                                                            "direction" : "forward",
                                                                            "indexBounds" : {
                                                                                   "_key" : [
                                                                                   "[\"posts:pid\", \"posts:pid\"]"
                                                                                   ],
                                                                                   "value" : [
                                                                                   "[MaxKey, MinKey]"
                                                                                   ]
                                                                            }
                                                                    }
                                                            }
                                                    }
                                            }
                                    }
                            }
                    ]
            },
            "serverInfo" : {
                    "host" : "085323710f1b",
                    "port" : 27017,
                    "version" : "3.2.7",
                    "gitVersion" : "4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2"
            },
            "ok" : 1
    }
    
    > db.objects.find({_key: 'topics:tid'}, {_id: 0, value: 1, score: 1}).noCursorTimeout().sort({score: 1, value: 1}).explain()
    {
            "queryPlanner" : {
                    "plannerVersion" : 1,
                    "namespace" : "0.objects",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                            "_key" : {
                                    "$eq" : "topics:tid"
                            }
                    },
                    "winningPlan" : {
                            "stage" : "PROJECTION",
                            "transformBy" : {
                                    "_id" : 0,
                                    "value" : 1,
                                    "score" : 1
                            },
                            "inputStage" : {
                                    "stage" : "SORT",
                                    "sortPattern" : {
                                            "score" : 1,
                                            "value" : 1
                                    },
                                    "inputStage" : {
                                            "stage" : "KEEP_MUTATIONS",
                                            "inputStage" : {
                                                    "stage" : "SORT_KEY_GENERATOR",
                                                    "inputStage" : {
                                                            "stage" : "FETCH",
                                                            "inputStage" : {
                                                                    "stage" : "IXSCAN",
                                                                    "keyPattern" : {
                                                                            "_key" : 1,
                                                                            "value" : -1
                                                                    },
                                                                    "indexName" : "_key_1_value_-1",
                                                                    "isMultiKey" : false,
                                                                    "isUnique" : true,
                                                                    "isSparse" : true,
                                                                    "isPartial" : false,
                                                                    "indexVersion" : 1,
                                                                    "direction" : "forward",
                                                                    "indexBounds" : {
                                                                            "_key" : [
                                                                                   "[\"topics:tid\", \"topics:tid\"]"
                                                                            ],
                                                                            "value" : [
                                                                                   "[MaxKey, MinKey]"
                                                                            ]
                                                                    }
                                                            }
                                                    }
                                            }
                                    }
                            }
                    },
                    "rejectedPlans" : [
                            {
                                    "stage" : "PROJECTION",
                                    "transformBy" : {
                                            "_id" : 0,
                                            "value" : 1,
                                            "score" : 1
                                    },
                                    "inputStage" : {
                                            "stage" : "SORT",
                                            "sortPattern" : {
                                                    "score" : 1,
                                                    "value" : 1
                                            },
                                            "inputStage" : {
                                                    "stage" : "KEEP_MUTATIONS",
                                                    "inputStage" : {
                                                            "stage" : "SORT_KEY_GENERATOR",
                                                            "inputStage" : {
                                                                    "stage" : "FETCH",
                                                                    "inputStage" : {
                                                                            "stage" : "IXSCAN",
                                                                            "keyPattern" : {
                                                                                   "_key" : 1,
                                                                                   "score" : -1
                                                                            },
                                                                            "indexName" : "_key_1_score_-1",
                                                                            "isMultiKey" : false,
                                                                            "isUnique" : false,
                                                                            "isSparse" : false,
                                                                            "isPartial" : false,
                                                                            "indexVersion" : 1,
                                                                            "direction" : "forward",
                                                                            "indexBounds" : {
                                                                                   "_key" : [
                                                                                   "[\"topics:tid\", \"topics:tid\"]"
                                                                                   ],
                                                                                   "score" : [
                                                                                   "[MaxKey, MinKey]"
                                                                                   ]
                                                                            }
                                                                    }
                                                            }
                                                    }
                                            }
                                    }
                            }
                    ]
            },
            "serverInfo" : {
                    "host" : "085323710f1b",
                    "port" : 27017,
                    "version" : "3.2.7",
                    "gitVersion" : "4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2"
            },
            "ok" : 1
    }
    

    "I want you to sort this list by the first column." "Ok, I'll sort the list by the first column."

    "I want you to sort this list by the first column, and then for things that are the same in the first column, compare the second column." "Ok, I'll load the entire database into memory and then sort it."


  • Discourse touched me in a no-no place

    @ben_lubar said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    "I want you to sort this list by the first column, and then for things that are the same in the first column, compare the second column." "Ok, I'll load the entire database into memory and then sort it."

    So it's webscale only if you have a very small web?


  • Discourse touched me in a no-no place

    @ben_lubar Remember what Blakey said about temp tables?



  • @FrostCat said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    @ben_lubar Remember what Blakey said about temp tables?

    That doesn't work before you get the data into the database that supports temp tables.


  • Discourse touched me in a no-no place

    @ben_lubar said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    That doesn't work before you get the data into the database that supports temp tables.

    Geez, Ben, Blakey even told you how to deal with that: read the shit in without sorting. Inserts will build the index. You can filter after that.

    I've been doing that for like 20 years.



  • @FrostCat said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    Inserts will build the index.

    There already is an index on the column it's sorting by.


  • Discourse touched me in a no-no place

    @ben_lubar said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    There already is an index on the column it's sorting by.

    Well then there's something wrong. But by all means don't try not sorting because there's obviously no way that could work.



  • @Arantor was the only TDWTF forum member to have posted over 10000 posts in 1 year.

    fake edit: But then @boomzilla did it too.

    fake edit: FUCKING HELL @accalia HIT 12345k

    fake edit: @abarker hit 10k as well

    fake edit: goddamnit @accalia stop making so many posts in the past you hit 16789k I mean 20k

    fake edit: a wild @RaceProUK has appeared!

    fake edit: @Onyx has entered the battle!

    fake edit: hi @FrostCat

    fake edit: @accalia hits 22222 posts per year. The top 20 posters each have more than double the top poster from any time on CS

    fake edit: @blakeyrat hits 10k. every user that ever had more than 10k posts per year has never dropped below that level.

    fake edit: @accalia hits 25k. :trophy: @Arantor drops below 10k. :cry:

    fake edit: :coffin: @Onyx


  • Discourse touched me in a no-no place

    @ben_lubar said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    fake edit: hi @FrostCat

    What, the inaugural spoon holder doesn't get to know what his final count was?


  • Discourse touched me in a no-no place

    @ben_lubar said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    The top 20 posters each have more than double the top poster from any time on CS

    I had--I believe--fewer than 640 posts on CS. In like 10 years.



  • @FrostCat said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    What, the inaugural spoon holder doesn't get to know what his final count was?

    0_1467972532708_upload-6f015d8f-b4a8-44a8-a243-2a0e4907e44b

    Your welcome.


  • Discourse touched me in a no-no place



  • @FrostCat You tell me. Its you'res.


  • Discourse touched me in a no-no place

    @boomzilla said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    You tell me. Its you'res.

    It may be, but I don't know to what you're referring.



  • @FrostCat Apparently your penchant for giving your password to blakey.


  • Winner of the 2016 Presidential Election


  • Discourse touched me in a no-no place

    @boomzilla said in @ben_lubar tries to parse <del>ServerCooties</del> data:

    Apparently your penchant for giving your password to blakey.

    That implies that you're me.


  • Discourse touched me in a no-no place

    @error mmmm, bacon.



Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.