Is MongoDB for me? (No, it's not, this is now @mott555's Random Node.js Question Thread)
-
As you may remember from my posts in the Status thread a while back, my home ESXi server committed seppuku through some unexplainable RAID-1 corruption that destroyed all my data. Among other things, I lost my personal SVN server which had the web-based MUD I was developing. I discovered I have an old working copy so all is not lost, but decided it was time to re-examine my goals and perhaps start from scratch (yet again) now that I'm much more comfortable with Node.js.
I want to stick with Node.js since I actually really like that platform for this kind of project. Previously I used SQLite. That worked fairly well and relational databases are second-nature to me, but there was a lot of boilerplate CRUD code to map database tables to JavaScript classes.
Are any of you guys MongoDB gurus? I want to see if it's worth switching to. It seems to store everything natively as JSON which should mean I have absolutely no ORM layer to deal with.
Does MongoDB have any concept of foreign keys? For example, I'll have Rooms, Players, NPCs, Items, etc. all linked together through various relationships to define what's where, how many are there, and such. It seems like MongoDB just keeps a big object graph, so how would that deal with potential object cycles? Or is it smart enough for that not to be a problem?
I'm also open to any suggestions for guides or tutorials that can help me understand Mongo without getting the SQL lobes of my brain all twisted up in a knot. I've skimmed tutorials and docs but it looks so foreign to me with my SQL relational-database-colored glasses.
-
I want to stick with Node.js since I actually really like that platform
Does MongoDB have any concept of foreign keys? For example, I'll have Rooms, Players, NPCs, Items, etc. all linked together through various relationships to define what's where, how many are there, and such. It seems like MongoDB just keeps a big object graph, so how would that deal with potential object cycles? Or is it smart enough for that not to be a problem?
you are thinking like a relational database person again. mongodb stores documents. that's it.i don't think mongo is for you.
try this instead:
That's what we went for with SockRPG and it's working well for us. bonus points for having multiple RDBMS support out of the gate.
-
NoSQL doesn't make sense for anything smaller than the top 1000 websites in the world.
Then there are nosql databases that lose data in trivial situations like MongoDB. Look at Aphyr's call-me-maybe tests for MongoDB: https://aphyr.com/posts/284-call-me-maybe-mongodb
Cassandra is much, much faster than it, and doesn't lose data so trivially. Answer is: MongoDB isn't good for anyone.
-
Pretty much every NoSQL database works the same way - you put JSON or whatever in and give it an identifier string. They mostly differ with how they handle indexing. I think MongoDB has a more traditional type of index where you mark some field of the JSON document as "I want to search based on this". I haven't used MongoDB much.
Redis doesn't have documents. Instead, it has types like "sorted set", which allows you to have a set of strings with numbers attached to them, so you can use that for email address -> user ID or something similar.
Couchbase has map/reduce indexes. You write a JavaScript function that does something to the data and then you can query the result of that function.
-
Answer is: MongoDB isn't good for anyone.
I came here to say exactly this. So ... QFT ... and I'll show myself out ;)
-
Maybe I should just stick with SQLite then, and try to find a better way to deal with the boilerplate CRUD code. I originally used it because it was in-process and used an npm module, no need for game admins to actually set up and manage a database instance at all, and that's probably still worth the development overhead I was dealing with.
-
Are we having second thoughts about migrating to NodeBB?
-
In theory a postgresql driver can be written for it.
@swayde, you're thinking of writing a MariaDB adapter for our DBAL? That's pretty cool; I'd be happy to help point you in the right direction or even lend a hand if you are.
btw, greetings from nodebb
-
Are we having second thoughts about migrating to NodeBB?
I ... never ... migrated ... to that ... o_O [me is confused]
-
Does MongoDB have any concept of foreign keys?
No. No keys at all. No joins, really, although you can fudge those. It does have indexes.
It does have auto-generated (if you don't provide one) GUID IDs, which I guess you can think of as a "primary key" if that helps you.
MongoDB records are "documents" in a JSON variant they call BSON, up to 16 MB in size. The expectation is that every entity you're storing fits in a single document. This is great if you're, say, Twitter. Not so great if you're, say, Amazon-- since it doesn't take a lot of Amazon shopping to get a purchase history over 16 MB in size.
For example, I'll have Rooms, Players, NPCs, Items, etc. all linked together through various relationships to define what's where, how many are there, and such. It seems like MongoDB just keeps a big object graph, so how would that deal with potential object cycles?
What I'd do is make three collections, one for rooms, one for MOBs (players and NPCs), one for player login/auth data.
Rooms can refer to the entities located in them using the BSON IDs of the entities. (Basically, your JSON would look like: "room_contains: [ 4324-fds-3241, 32414-dsa-432425, etc ]).
Since you don't have joins, you need to do all these lookups yourself in code. Fortunately, unless you have the Universe's Largest MUD, it should be no problem to keep the entire MUD world in memory at all times. Oh, BTW: unless things have changed very recently, MongoDB also doesn't have transactions-- so if you store a document that relies on another document, be aware you can easily get into an inconsistent state.
MongoDB has basically none of the ACID guarantees of a real SQL server. (I'm not sure if SQLite has them, either. But if you code to SQLite you can swap it for Postgres, or MS SQL, or whatever later on. If you code to MongoDB, you're kind of married to it forever.)
Also keep in mind the 16 MB document limit. Especially for players who collect items, and each item could have custom descriptions/stats.
And I said it above, but especially keep in mind that MongoDB is designed with the assumption that the entity you're storing fits (self-contained) in a single 16 MB document. That assumption is at the very core of the product.
-
MongoDB is only for you if you are a mong.
-
Answer is: MongoDB isn't good for anyone.
I just read an overview of MongoDB and it sounds dumb and useless to me...is there really any advantage to it at all?
-
I just read an overview of MongoDB and it sounds dumb and useless to me...is there really any advantage to it at all?
Like I said, it has very very specific use-cases, where you're storing mostly stand-alone entities that are small in size, where you plan to grow in a widely distributed fashion by making tons of replicants or shards, where it's no big deal if one of those entities disappears into the ether.
So: Facebook timeline items, Twitter statuses, etc.
That's about all its good for. And I think the consensus is that it's not even as good at that as its competitors are.
-
Sounds like MongoDB is a very poor fit for me then.
-
Ah. I suppose my assessment was correct, then (as I don't personally plan on writing a Facebook/Twitter clone any time soon). Back to MySQL, I suppose!
-
I just read an overview of MongoDB and it sounds dumb and useless to me...is there really any advantage to it at all?
For MongoDB specifically, node developers like it because it replaces SQL with javascript. And it is easier to use as compared to other NoSQL databases. It has a lot of bugs and performance issues, and is far slower than postgresql if it's running in a single computer.
NoSQL in general makes differente choices in the CAP theorem. It has better partition tolerance and worse consistency. You can read and write data hiting only a few of the servers in a cluster. (It doesn't make sense for a single server).
When you read data from a normal SQL database, it has to ensure that you're reading updated commited data. To assure you of it, the server you're querying needs to be informed of any write that happened at the other servers. This have a high cost on partition tolerance.
-
Okay, I'm sticking with SQLite.
It took a bit for the whole concept of Promises to percolate through my brain. For a bit, I ended up in callback hell with Promises nested within Promises!
I get it now though, and holy smokes my basic CRUD code LOC was reduced to about 20% of the original toxic hellstew of nested callbacks and async.series arrays!
-
original toxic hellstew of nested callbacks and async.series arrays!
I inherited something like that. It uses an MVC + API + Repository collection + Factory + Entity Framework.
What should have been a straight call to an API endpoint ended up being a hacked-together call to an MVC controller, which has instantiated an API-interface that constructs a call the api (itself) from the interface factory, which calls the api, which instantiates the correct repository object, which calls the database,, and eventually returns the data all the way back up the pipe so it can be (somewhat) manually parsed into JSON.What are we talking about again?
-
For MongoDB specifically, node developers like it because it replaces SQL with javascript.
Ah, so my suspicion was probably correct, then. It's the latest generation of developers crying, "WAAAH! I DON'T WANNA LEARN SQL!"
-
If you're doing something RPG-like, an RDBMS is going to be a natural fit for all the complex relationships between entities that an RPG entails, like item affixes, monster ability damage, etc.
-
Yeah. That's my impression.
We have a VP who is really into the 'in the trenches' type stuff. He was trying to convince me that Mongo would be a great replacement for SQL Server for our mission critical databasing needs.
Nope, sorry buddy. If we drop even a single row of data it's compliance penalties bad. Plus almost everything we do is relational anyway.
Similarly, I have a developer on my team who is really into the 'USE ALL THE SHINY NEW THINGS' type stuff. He was trying to convince me that Mongo would be a great supplement to SQL Server for all our nonrelational databasing needs.
Nope, we'll keep using a SQL Server table with one column for that, because we STILL can't lose data without getting penalized out the ass.
-
What are we talking about again?
A Discourse clone written in Java, by your description.
-
Ah, so my suspicion was probably correct, then. It's the latest generation of developers crying, "WAAAH! I DON'T WANNA LEARN SQL!"
MongoDB was created to solve a particular problem, and it did solve that problem... perhaps it was not the best solution for that problem but it did solve it.
Then some lazy developers found it and said: "I can use a database and don't have to worry about learning SQL? i'm having me some of that!" and so they took MongoDB into an alleyway, hit it over the head with a blackjack, mugged it, tied it up and forced it to work for them.
and MongoDB worked for them, after a fashion, It was slow, it was crippled, it was entirely the wrong solution... but it mostly worked.
The lazy developers then shouted to the world "Look at this wonderful new database i have found! see how it frees us from having to think about out data structure and lets us just code! see how awesome i am for discovering it and sharing it with you!"
and lo, MongoDB spread. It was still crippled, and on dark evenings, hidden in the quiet hum of data center HVAC after all but the most essential of night shift workers (the single janitor) had left for the day one could sometimes hear the sobbing of MongoDB, chained, forced to labor to solve a problem it was not designed to solve, yearning to be free once more.
But freedom was not to be had...
-
wipes a single tear
That was... beautiful.
-
Depending on the size of your "active" data, you might have the situation similar to mine. As in, you can just keep all the data in node.js memory and occasionally dump to disk, for persistence. Kind of like save game / load game in traditional games.
Of course, that'll limit the number of active games you can handle, so that'd be the downside.
As for the SQL hell problem, I'm personally partial to this solution: https://github.com/brianc/node-sql
-
-
I cant think of a reason to use it instead of Cassandra. I mean, it is easier to use, but if you need that level of performance its worth it.
Cassandra lacks consistency, but you can make it ensure that all data is written to multiple servers before returning an ok status.
Disclaimer: I never used Cassandra. I used MongoDB for a small project and had performance problems. Slow deletes and a lack of the skip sql command I would use for paging. NodeBB seems to have solved the paging problem, there are probably workarounds for all serious issues. Noone sugested a workaround for the delete problem when I asked around. People said me nosql dbs are normally slow at deleting.
-
I'm not sure if SQLite has them, either.
SQLite is ACID in everything (including DDL). It only allows one writer at a time for the whole database though and while it is generally very fast, commit is pretty slow. This makes it great for persisting data on the client, but poor match for most kinds of servers.
-
-
As in, you can just keep all the data in node.js memory and occasionally dump to disk, for persistence. Kind of like save game / load game in traditional games.
For the most part, this is what I'm doing. At application start, everything's loaded from the database into JavaScript data structures and almost everything stays there. Anytime something important changes, the changes are written back to the database for persistence, but the database itself is hardly ever read from after the server starts up.
-
Yup, SQL is an overkill here.
You can see my help thread for more details: https://what.thedailywtf.com/t/help-me-pick-a-persistence-solution/51875
I went with the SQLite used as a KV store. So, mostly, there's just one table, with keys and json data. Load at start, save when needed, as you described.
I get the occasional "SQLite is busy" event, but nothing too bad. It serves its purpose.
-
So, question about Promises. How the flip do I do unit testing on stuff with promises? If I have assertions within a
.then()handler, and the assertions fail, the promise library (I'm using bluebird) swallows the exceptions and line numbers and gives me a general error that doesn't let me know which assertion failed, which makes the unit test very difficult to correct when something's wrong.Is there a way to not swallow assertions?
-
Nevermind, there is a line number for the failed assertion, it just wasn't blindingly obvious.
-
How the flip do I do unit testing on stuff with promises?
If you're not already, use Sinon. You can very easily mock out methods that return promises and force them to resolve or reject for testing purposes.
-
You also probably want chai-as-promised if you're using mocha and chai. Don't mix callbacks and promises, that leads to headaches.
-
-
-
-
Fun Fact: Discourse inserted a bunch of line breaks into the previous post for no reason at all.
-
Fun Fact: Discourse inserted a bunch of line breaks into the previous post for no reason at all.
Does it to me all the time.
-
edit: NOREPRO
-
Does it to me all the time.
edit: NOREPRO
I think we have a dictionary definition of DiscoBug here.
-
Documentation for the bluebird promise library is a joke...
I have an issue I'm not sure how to handle. The server receives a command from a client and needs to process it. I have an array of command instances (each command is its own class) and want to process them serially, until a command reports that it applied and was successful. Each command is async now and returns a Promise when run.
What's the best way to loop over this array of commands? Promise.all is not the right thing because I want to deal with them all serially, not in parallel, and there are issues where commands need to be tried in a certain order because some commands may have similar or identical syntax and order-of-operations is important (I try to avoid this but there are a couple cases where it is unavoidable).
The important thing is not to try the next command until the current one is finished.
I suppose I could write a big ugly function like this:
commands[0].Try() .then(function() { return commands[1].Try(); }).then(function() { return commands[2].Try(); }).then(function() { return commands[3].Try(); }).then(function() { return commands[4].Try(); }).then(function() { return commands[5].Try(); });etc, but that doesn't look right and it'll be a big pain because I know I'll forget to update it each time I add a new command.
-
I forgot to say, in my old codebase each command was NOT async and this made this easy with a standard for loop. However I had other issues I was constantly working around as a result of using a synchronous command parser; if I can figure this out an async command parser will be much better in most ways.
-
brb, going to try something with recursion...if you don't hear back from me it's because my stack exploded and blew me up.
-
I think I got it working using recursion. Not gonna post the solution here though because it's long and absolutely hideous, however it works and I don't have to change anything other than add new commands to the command array.
-
Documentation for the bluebird promise library is a joke...
i won't claim that this library is any better in the documentation department but at least you know the developer to yell at her when it breaks.
you may find this library helpful.
GitHub - AccaliaDeElementia/async-as-promised: Utilities for dealing with Promises and turning callback based APIs into promises
Utilities for dealing with Promises and turning callback based APIs into promises - AccaliaDeElementia/async-as-promised
(which will need a name change because there's an abandonware project in NPM by the same name. most annoying.)
you can install with
npm install https://github.com/AccaliaDeElementia/aplus/archive/0.1.0.tar.gzto install the one and only release so far.in particular i think you'll be interested in
aplus.seriesexamples: https://github.com/AccaliaDeElementia/aplus/blob/master/test/lib/aplus-series.js
and of course i'd be happy to help!
-
Thanks, but I got something workable and I don't think that pattern will come up again in this project.
That said, my new command parser structure is awesome. It's actually unit-testable this time!
-
Thanks, but I got something workable and I don't think that pattern will come up again in this project.
true.
still there are alot of other helpful functions in there. poke around if you want inspiration for other design solutions you may encounter with promises. ;-)
-
I have an issue I'm not sure how to handle. The server receives a command from a client and needs to process it. I have an array of command instances (each command is its own class) and want to process them serially, until a command reports that it applied and was successful. Each command is async now and returns a Promise when run.
What's the best way to loop over this array of commands? Promise.all is not the right thing because I want to deal with them all serially, not in parallel, and there are issues where commands need to be tried in a certain order because some commands may have similar or identical syntax and order-of-operations is important (I try to avoid this but there are a couple cases where it is unavoidable).
The important thing is not to try the next command until the current one is finished.
I did something like that with this recursive function:
var APIs = [WiktionaryAPI, UrbanDictionaryAPI, WikipediaAPI, GoogleCustomSearchAPI]; (function loop() { var API = APIs.shift(); if (API) { console.log("Using " + API.name + "..."); API(arg, function callback(match) { if (match) { // this one worked, do the needful thing console.log("Found: " + match); } else { // didn't work, proceed to the next console.warn(API.name + " returned nothing"); loop(); } }); } else { // all were tried, nothing worked } })();
