MongoDB? What's that, some kind of fruit? Give me a real database!


  • Garbage Person

    Re: notifications are actually slow for reals

    Alright, so I'm going to take a look at making an RDBMS connector for NodeBB. Because at some point I should probably figure Node out because it seems to be growing in importance, even in Microsoftlandia.

    Probably Postgres.

    I may also borrow some techniques from WtfFramework and actually allow automatic creation of database structures for complex objects. It's super-fucking slick and covers the vast majority of compelling reasons for using document store DBs anyway. If that works out, I have some awesome automatic indexing techniques that work in MSSQL - if PG has as much self-examination capability, no reason it couldn't be done there, too.

    If it works out the way I think it will, I'll also do MSSQL because I am a perverted asshole. like ruining opensource projects with proprietary enterprise M$ software hate Oracle am awesome.

    I don't Node. I don't Javascript. I don't Mango. I don't even Git. I am a Microsoft ecosystem Enterprise developer. I am a masochist and love torturing myself in public. In the spirit of the WtfCorp lounge thread, in this thread effectively liveblog my efforts to teach the Webscale set how it's done.

    I stuffed this in General because it's not WTF (well, yet, anyway). It's not Lounge secure. It's not meta. So it must be General. Unless I'm being an AW....


    Chapter 1: I need a development environment.
    Well, may as well get a test instance first and build a development toolchain out of that.
    Let's see here.... www.google.com nodebb clicky. Okay. I don't want to pay for it so... Community? Other apps call the free version Community Edition, so that sounds like a good start. clicky
    Shit, no, that's a forum.

    Pricing? Huh. I mean, that's some money I guess. Ah, there we are. Developer's Edition. Stuffed in the corner.
    Interesting how there's not a reference to any free production installs. I wonder if the license prohibits it or something.
    clicky

    WHY AM I ON GITHUB!? The Developer's Edition is just the source!? Aw hell.
    There we go. Readme.md has a pointer to installation instructions.

    SWEET FUCK IT SUPPORTS WINDOWS!? What is this sorcery?

    Alright, lemme follow some directions....


  • FoxDev


  • Garbage Person

    @RaceProUK I don't trust ORMs with performance-critical database code. Mostly because they cannot be trusted. Sure, your average CRUD configuration application will be fine, but anything with large quantities of data or large speed expectations (and a human communication tool is definitely in the 'should be fast' category) doesn't do well. I've already learned that lesson the hard way 🐈


  • Garbage Person

    Assuming this Windows install actually works out, I'm going to pivot and do MSSQL first.

    My reasons are twofold:

    1. I am more confident in my ability to quickly produce a viable performant product
    2. I am a perverted SOB

    This has unfortunate implications for the potential immediate usefulness for WTDWTF, but you work with the developers you have, not the developers you want.

    Fortunately, once I've got a baseline schema in place and working, it shouldn't be hard to branch out to the other SQLs if you leave out all the MSSQL-specific performance stuff I have in mind.



  • @Weng 3. MS SQL is getting a Lunix port soon anyway. So loser Lunix people can run it on Lunix. Losers.

    If you just directly (automagically) adapt a DB layer intended for a NoSQL database into MS SQL, you'll end up with really wide inefficient tables. Because you'll be able to add columns as is necessary, but you'll have no provision for removing them afterwards when they become obsolete.


  • Garbage Person

    Chapter 2: I am Blakeyrat!

    C:\Users\Weng\Documents\GitHub\NodeBB-SQL\nodebb [v1.x.x]> npm install
    Error: ENOENT, stat 'C:\Users\Tony\AppData\Roaming\npm'

    WTF?

    google
    WTF!

    Okay, so...
    C:\Users\Weng\Documents\GitHub\NodeBB-SQL\nodebb [v1.x.x]> npm install
    Error: ENOENT, stat 'C:\Users\Weng\AppData\Roaming\npm'

    C:\Users\Weng\Documents\GitHub\NodeBB-SQL\nodebb [v1.x.x]> mkdir C:\Users\Weng\AppData\Roaming\npm

    Directory: C:\Users\Tony\AppData\Roaming

    Mode LastWriteTime Length Name


    d---- 3/31/2016 9:35 PM npm

    C:\Users\Weng\Documents\GitHub\NodeBB-SQL\nodebb [v1.x.x]> npm install
    {shit proceeds normally from here}

    EXCUSE ME, NPM. How the fuck are you not running in the security context of the powershell prompt I launched you from?

    Also, would "Error: No Entry Permissions for C:\Users\Weng\Appdata\Roaming\npm" have been too fucking hard? ENOENT? Really?


  • Garbage Person

    @blakeyrat Oh absolutely. The mainline schema won't be handled like that at all. Any automagic bolt-on wide-tables stuff will be plugin-oriented.

    Hey look at that.

    31/3 21:52 [12312] - warn: NodeBB Setup Aborted.
    MongoError: E11000 duplicate key error collection: 0.objects index: key_1_valu
    e
    -1 dup key: { : "config", : null }

    Fixed by... Blowing out the MongoDB data directory and retrying.



  • mongo in pt-br is also a slang for retard. retard-db is a pretty appropriate name for it



  • This post is deleted!

  • Garbage Person

    Hey guys, I have a forum! It was faster to install than that time I tried to do Discourse!

    0_1459476429354_upload-32dd2ab3-4740-4f61-a166-e93f129d8edc


  • Garbage Person

    Why the hell do all opensource databases forbid official GUI management tools?

    0_1459476716356_upload-19e9ca4c-6262-43da-8c16-704c780320c7

    Whatever. There's no way it takes me more than 30d to figure out a schema.


  • Fake News

    http://www.youtube.com/P8ciVBQixpU
    Candygram for Mongo – 00:42
    — zfestini

    The fuck, NodeBB won't auto-embed videos?

    [You have to have a correct URL. Even youtube doesn't work with the one you used - bz]


  • Garbage Person

    Chapter 4: A tour of the Mango

    Righto. So let's dive right in here.... Any place is as good as any to start.

    {
    "_id" : ObjectId("56fdd5e08fcd1f9e87fcd115"),
    "_key" : "cid:0:children",
    "value" : "3",
    "score" : 4
    }

    Errrr. We'll come back to this. Let's try starting with the oldest objects first.

    { "_id" : ObjectId("56fdd5e08fcd1f9e87fcd0e2"), "_key" : "config", "title" : "NodeBB", "showSiteTitle" : 1, "postDelay" : 10, "initialPostDelay" : 10, "newbiePostDelay" : 120, "newbiePostDelayThreshold" : 3, "minimumTagsPerTopic" : 0, "minimumPostLength" : 8, "maximumPostLength" : 32767, "maximumTagsPerTopic" : 5, "allowLocalLogin" : 1, "allowGuestSearching" : 0, "allowTopicsThumbnail" : 0, "allowAccountDelete" : 1, "registrationType" : "normal", "allowFileUploads" : 0, "allowUserHomePage" : 1, "maximumFileSize" : 2048, "minimumTitleLength" : 3, "maximumTitleLength" : 255, "minimumUsernameLength" : 2, "maximumUsernameLength" : 16, "requireEmailConfirmation" : 0, "minimumPasswordLength" : 6, "maximumSignatureLength" : 255, "maximumAboutMeLength" : 1000, "maximumProfileImageSize" : 256, "maximumCoverImageSize" : 2048, "profileImageDimension" : 128, "allowProfileImageUploads" : 1, "teaserPost" : "last", "allowPrivateGroups" : 1, "theme:type" : "local", "theme:id" : "nodebb-theme-persona", "theme:staticDir" : "", "theme:templates" : "", "theme:src" : "" }

    Alright. This is recognizable. Fuck, it's the kind of thing that NoSQL was born for. Key value pairs! It also appears to be quite a lot of the configuration.

    {
    "_id" : ObjectId("56fdd5e08fcd1f9e87fcd0e3"),
    "_key" : "plugins:active",
    "value" : "nodebb-theme-persona",
    "score" : 0
    }

    Okay. So using my ESP, key is going to be a heirarchical organizational structure, value is going to be the value. No clue what score is yet.

    {
    "_id" : ObjectId("56fdd5e08fcd1f9e87fcd0e4"),
    "_key" : "global",
    "nextCid" : 4,
    "nextUid" : 2,
    "userCount" : 2,
    "nextTid" : 1,
    "topicCount" : 1,
    "nextPid" : 3,
    "postCount" : 3,
    "nextChatRoomId" : 1,
    "nextMid" : 1,
    "uniqueIPCount" : 1
    }

    Global counters.... These will become sequences or identity columns as appropriate.

    { "_id" : ObjectId("56fdd5e08fcd1f9e87fcd0e5"), "_key" : "category:1", "cid" : 1, "name" : "Announcements", "description" : "Announcements regarding our community", "icon" : "fa-bullhorn", "bgColor" : "#fda34b", "color" : "#fff", "slug" : "1/announcements", "parentCid" : 0, "topic_count" : 0, "post_count" : 0, "disabled" : 0, "order" : 1, "link" : "", "numRecentReplies" : 1, "class" : "col-md-3 col-xs-6", "imageClass" : "cover", "descriptionParsed" : "Announcements regarding our community" }

    Blam. Category object. Maybe some persisted computed columns in topic_count/post_count.

    { "_id" : ObjectId("56fdd5e08fcd1f9e87fcd0e6"), "_key" : "categories:cid", "value" : "1", "score" : 1 } My ESP tells me "cid" is "category ID". Why is this an object by itself? What is score? Why isn't it just a property of the category?

    ... Wait, can you access an object in Mango without pulling back all of its properties? Is this a homegrown index?

    { "_id" : ObjectId("56fdd5e08fcd1f9e87fcd0e7"), "_key" : "cid:0:children", "value" : "1", "score" : 1 } Category 0 apparently has 1 children? Not sure what that child would be. There aren't any immediately apparent on the forum itself.

  • Garbage Person

    This one's interesting:

    {
    "_id" : ObjectId("56fdd5e08fcd1f9e87fcd0e8"),
    "_key" : "groups:createtime",
    "value" : "cid:1:privileges:groups:find",
    "score" : 1459475936283.0
    }

    First reference to an object heirarchy in a value. That makes it a relationship. REALLY have no clue WTF score is. Also unclear is what this relationship actually represents.


  • Garbage Person

    Okay. Let's take a different approach. I have figured out how to get a list of all the distinct _key values.

    [ "analytics:pageviews", "analytics:pageviews:byCid:1", "analytics:pageviews:byCid:2", "analytics:pageviews:month", "analytics:posts", "analytics:posts:byCid:2", "analytics:uniquevisitors", "categories:cid", "category:1", "category:2", "category:3", "category:4", "chat:room:1", "chat:room:1:uids", "cid:0:children", "cid:1:read_by_uid", "cid:2:pids", "cid:2:read_by_uid", "cid:2:tids", "cid:2:tids:posts", "cid:2:uid:1:tids", "config", "email:sorted", "email:uid", "global", "group:Global Moderators", "group:administrators", "group:administrators:members", "group:administrators:owners", "group:cid:1:privileges:groups:find", "group:cid:1:privileges:groups:find:members", "group:cid:1:privileges:groups:read", "group:cid:1:privileges:groups:read:members", "group:cid:1:privileges:groups:topics:create", "group:cid:1:privileges:groups:topics:create:members", "group:cid:1:privileges:groups:topics:reply", "group:cid:1:privileges:groups:topics:reply:members", "group:cid:2:privileges:groups:find", "group:cid:2:privileges:groups:find:members", "group:cid:2:privileges:groups:read", "group:cid:2:privileges:groups:read:members", "group:cid:2:privileges:groups:topics:create", "group:cid:2:privileges:groups:topics:create:members", "group:cid:2:privileges:groups:topics:reply", "group:cid:2:privileges:groups:topics:reply:members", "group:cid:3:privileges:groups:find", "group:cid:3:privileges:groups:find:members", "group:cid:3:privileges:groups:read", "group:cid:3:privileges:groups:read:members", "group:cid:3:privileges:groups:topics:create", "group:cid:3:privileges:groups:topics:create:members", "group:cid:3:privileges:groups:topics:reply", "group:cid:3:privileges:groups:topics:reply:members", "group:cid:4:privileges:groups:find", "group:cid:4:privileges:groups:find:members", "group:cid:4:privileges:groups:read", "group:cid:4:privileges:groups:read:members", "group:cid:4:privileges:groups:topics:create", "group:cid:4:privileges:groups:topics:create:members", "group:cid:4:privileges:groups:topics:reply", "group:cid:4:privileges:groups:topics:reply:members", "group:registered-users", "group:registered-users:members", "groups:createtime", "groups:visible:createtime", "groups:visible:memberCount", "groups:visible:name", "groupslug:groupname", "ip:127.0.0.1:uid", "ip:recent", "message:1", "navigation:enabled", "notifications", "notifications:new_post:tid:1:pid:3:uid:2", "plugins:active", "post:1", "post:2", "post:3", "posts:pid", "schemaDate", "settings:emoji-extended", "settings:emoji-extended-sets", "tid:1:followers", "tid:1:posts", "tid:1:posts:votes", "topic:1", "topics:posts", "topics:recent", "topics:tid", "topics:views", "uid:1:chat:room:1:mids", "uid:1:chat:rooms", "uid:1:chat:rooms:unread", "uid:1:followed_tids", "uid:1:ip", "uid:1:notifications:unread", "uid:1:posts", "uid:1:tids_read", "uid:1:topics", "uid:1sessionUUID:sessionId", "uid:2:chat:room:1:mids", "uid:2:chat:rooms", "uid:2:ip", "uid:2:posts", "uid:2:sessions", "uid:2:tids_read", "uid:2sessionUUID:sessionId", "user:1", "user:1:settings", "user:2", "user:administrators:settings", "user:guests:settings", "user:registered-users:settings", "username:sorted", "username:uid", "users:joindate", "users:notvalidated", "users:online", "users:postcount", "users:reputation", "userslug:uid", "widgets:global" ]

    I have 1 chat room.
    4 categories.
    2 users.
    The default groups (global moderators, administrators, registered-users)
    3 posts
    2 users

    Add in some system tables, and this actually makes plenty of sense right out of the gate.

    There's also a separate sessions collection that contains the usual stuff you'd find in a session. So that gets a table.

    Now I'll go away and draw a first pass ERD.


  • Garbage Person

    Breakthrough: "Score" is used for a numeric value.

    For instance:

    {
    "_id" : ObjectId("56fddf808fcd1f9e87fcd1b6"),
    "_key" : "analytics:pageviews:byCid:1",
    "value" : "1459476000000",
    "score" : 1
    }

    CategoryID 1 was last viewed at timestamp {value}, and has been viewed a total of {score} times.



  • @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    Wait, can you access an object in Mango without pulling back all of its properties? Is this a homegrown index?

    you can select just the properties you want, and it has indexes that are kind of the thing we have on sql, I dont remember the details



  • @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    Okay. So using my ESP, key is going to be a heirarchical organizational structure, value is going to be the value. No clue what score is yet.

    _key is the ID of the object (string). value is used for sorted sets (string). Sorted sets span multiple documents, one per entry. score is the sorting key for sorted sets, which is usually a timestamp but can be any number.


  • Garbage Person

    None of this is particularly hard so far...
    0_1459481900506_upload-7f103460-9225-4bb0-bf46-bcb9fb781500



  • @Weng it's not the last post timestamp, it's the timestamp for that data point in the analytics.


  • Garbage Person

    @ben_lubar Hm, so sorted sets are basically indexes? And therefore I can ignore them with impunity (except when they're being used to store actual meaningful data, at which point they need to get normalized into the real world)


  • Garbage Person

    @ben_lubar I'm just guessing at field semantics for now. Analytics is definitely way off base.



  • @Weng mostly indexes and analytics/tracking type stuff.


  • Winner of the 2016 Presidential Election

    @Weng In general, I agree RE: ORMs. SQLAlchemy (Python) is pretty cool, though, it lets you express a lot of queries without forcing you to use raw SQL (and doesn't do anything stupid).


  • Garbage Person

    @asdf Yeah, you can keep the expressiveness and composability, but you still lose things like control over lock strategy and index use that are key to wringing every last ounce of performance out.



  • Scores are present in sorted sets.

    So a set is like a hash table, key values. (in redis terminology anyway), and a sorted set is like a table with the score being the column we execute ORDER BY against.


  • Garbage Person

    @julianlam Gotcha.

    Looking at the JS a bit. Mapping this onto SQL is going to be interesting. I'm going to have to parse the key to pick tables and filters. This makes it easy, however, to leave the less critical parts in a document store.



  • @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    Why the hell do all opensource databases forbid official GUI management tools?

    I don't think they forbid(?) them, they're just too fucking lazy to make one.

    @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    Alright. This is recognizable. Fuck, it's the kind of thing that NoSQL was born for. Key value pairs!

    Mongo supports full JSON, including arrays and sub-objects. That said, since this forum supports Redis (which IIRC is just a dumb key/value store) they probably aren't using those. Probably.

    @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    ... Wait, can you access an object in Mango without pulling back all of its properties?

    Sure, why not? Also, Mongo stores "documents" not "objects".

    @ben_lubar said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    _key is the ID of the object (string). value is used for sorted sets (string). Sorted sets span multiple documents, one per entry. score is the sorting key for sorted sets, which is usually a timestamp but can be any number.

    It's SUCH a stupid idea to use Mongo like Redis. WTF is this shit. Someone put Captain Picard here.

    To make it compatible with Redis, you're basically using it in the LEAST efficient way possible. MONGO HAS ARRAYS ALREADY!


  • Winner of the 2016 Presidential Election

    @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    Why the hell do all opensource databases forbid official GUI management tools?

    pgAdmin?



  • @asdf I'm still confused about the use of the word "forbid". They're open source, how could they possibly "forbid" the use of a GUI tool?


  • Garbage Person

    @blakeyrat The business logic is unconfortably aware of persistence structure and has expectations. No object model.

    So the database API has to be catered to the lowest common denominator: redis.

    I'm going to build that back up, though.


  • Garbage Person

    @blakeyrat Hyperbole. I use it occasionally.



  • @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    @blakeyrat The business logic is unconfortably aware of persistence structure and has expectations. No object model.

    Of course not. Why would you use "abstraction" in a software product? Pfft.

    I've actually changed my mind. Seeing the weird deranged way they're using Mongo, I actually think MS SQL could get better performance for this workload.

    The joins'll be a bitch, although I suppose since they have no data layer, that's all done in business logic... puke.



  • @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    @blakeyrat Hyperbole. I use it occasionally.

    Now you have to explain how it's hyperbole. Huh? That adds confusion on top of the confusion.



  • @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    The business logic is unconfortably aware of persistence structure and has expectations. No object model.

    Meh.


  • Garbage Person

    @blakeyrat the interesting thing is that I suspect applying the same "unwind the redis style access into an object model access" technique could go a long way towards improving the state of things for Mongo mode.



  • @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    @blakeyrat the interesting thing is that I suspect applying the same "unwind the redis style access into an object model access" technique could go a long way towards improving the state of things for Mongo mode.

    No doubt!

    Sure you can use ANY DB system to simulate a key/value store, that's not difficult. But it's also using pretty much EVERY DB system other than Redis and its competitors in a broken inefficient way, and throwing the last 30 years of database theory in the trash.

    But! NodeBB is an open source product, so it's not like we expected it to have competent developers.


  • Garbage Person

    @blakeyrat Hyperbole is the exaggeration of statements, particularly for editorial effect. They don't forbid GUIs, but they're rare (and generally terrible when present). This annoys me and I felt like drawing extra attention to it.


  • Winner of the 2016 Presidential Election

    @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    and generally terrible when present

    Again, pgAdmin.


  • Garbage Person

    @blakeyrat On the other hand, NodeBB is an opensource product, so jerks like me can swoop in and fix it.

    Pretend we had a philosophical debate about whether users should have that burden here. I'm just doing this for professional practice unscrewing other people's messed up design and maybe to learn some things about stuff I don't get too play with in my present shithell job.



  • @Weng What's throwing me is that since they don't forbid GUIs, you're not using hyperbole, you're just saying something counter to reality. That's what's throwing me. You're "exaggerating" a thing that doesn't exist?

    Hyperbole would be something that exaggerates the case where the MongoDB team refused to make a GUI in the first place. Something like, "Bill from Mongo tried to make an official GUI once; they never found the body".



  • Just to answer the open sores DB tools point, if the DB is GPL, the connector library is also going to be GPL. The GPL loving people are all evil command line people.


  • Garbage Person

    @asdf I reserve judgement until I use it. The last time I did pg, pgadmin was only slightly less awful than the MySQL one.



  • @asdf said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    Again, pgAdmin.

    pgAdmin is definitely in the "pretty terrible" bucket. Unless it's REALLY improved in the last 2 years.


  • Garbage Person

    @blakeyrat I think there may be a peculiar kind of brain damage among DB people. They haven't used MSSQL and therefore haven't seen the glory that is SSMS (which admittedly has quirks) and are comparing against Toad or some shit.


  • ♿ (Parody)

    @blakeyrat said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    What's throwing me is that since they don't forbid GUIs, you're not using hyperbole, you're just saying something counter to reality. That's what's throwing me. You're "exaggerating" a thing that doesn't exist?

    Fuck where are the :whoosh: :badger:s when you need them.



  • @Weng I will say this for pgAdmin: it wasn't Java.

    For some reason their download page adds the Trademark symbol on Windows (correct but unnecessary), but omits the one on Mac OS X. Which is also spelled wrong. I thought open source-y developers liked Macs? Why the dig?


  • BINNED

    @Weng Still pretty shit. Well, I mean, the design is OK IMHO, it's easy to get to the data you need and get an overview of the stuff but god damn is it unstable. Also, likes to throw errors on larger chunks of pgSQL (regular SQL seems to be immune to this) a few times and then it magically starts working. Just pound that run button, sooner or later it will work! Sigh...

    pgAdmin aside, Postgres' JSON functions seem to work well and are quite performant if you want a temporary shim somewhere. I'd recommend 9.5 though, I'm pretty sure there are a few useful functions not present in 9.2 - 9.4 that just got added (can't remember which ones specifically but I remember missing them and being happy they added them).


  • Winner of the 2016 Presidential Election

    @Weng said in MongoDB? What's that, some kind of fruit? Give me a real database!:

    They haven't used MSSQL and therefore haven't seen the glory that is SSMS

    ^ This. Of the tools I've seen so far, it's definitely the least shitty one.


  • Garbage Person

    For my next trick, I install Visual Studio and SQL Server on a machine with no disk space free. Yay.


Log in to reply