Reputation vs. Posts
-
@anotherusername said in Reputation vs. Posts:
@FrostCat
08
is not a valid Octal number.I cannot rightly apprehend the confusion of etc., and so on.
-
@Lorne-Kates said in Reputation vs. Posts:
Why are we using it?
It was the least worse thing we could find, apparently. Were you sleeping?
-
-
@accalia said in Reputation vs. Posts:
that's why the only way i'm writing another shadowmod is if it has offline db access refreshed regularly. that way you can DOS shadowmod (by accident or malice) but you can't DOS the forum through that vector.
So, we finally found what "webscale" means :
you need a web-farm to copy your database over to it when you want to query it.
-
@boomzilla said in Reputation vs. Posts:
There are indexes. There are no fixed tables / columns over which to define and enforce relations.
Ah, so the retardery is slightly different from what I thought.
-
@boomzilla said in Reputation vs. Posts:
It was the least worse thing we could find, apparently.
Nobody's ever heard of Postgres? MySQL? MSSQL Express?
-
@FrostCat a leading 0 makes the number octal, but 8 isn't a valid octal digit. So it's not a valid number and doesn't get highlighted as a number.
Notice the
-
characters aren't highlighted either.
-
@FrostCat said in Reputation vs. Posts:
Nobody's ever heard of Postgres? MySQL? MSSQL Express?
No. I'm familiar with the DBs, but not the forums.
-
@TimeBandit said in Reputation vs. Posts:
you need a web-farm to copy your database over to it when you want to query it.
How big is the database again? The idea that you just can't do real-time queries on something in the tens of gigs...well, I can't quite come up with the right insult. I'm sure everyone who's ever written an ERP or payroll package is shocked.
-
@anotherusername said in Reputation vs. Posts:
a leading 0
No, I get that. I remember the ANSI edition of K&R, and everybody's favorite trivial change. I was boggling at the idea that that block of text should've been interpreting stuff as octal.
-
@boomzilla said in Reputation vs. Posts:
No. I'm familiar with the DBs, but not the forums.
Should I spell out that any of those would've been a better DB than Mongo? For more or less anything?
-
@FrostCat said in Reputation vs. Posts:
How big is the database again? The idea that you just can't do real-time queries on something in the tens of gigs...well, I can't quite come up with the right insult. I'm sure everyone who's ever written
an ERP or payroll packagesoftware using a real database engine is shocked.FTFY
-
@FrostCat said in Reputation vs. Posts:
Should I spell out that any of those would've been a better DB than Mongo?
Would that make you happy?
Where were all you people when we opened up the Migration category?
-
@TimeBandit said in Reputation vs. Posts:
you need a web-farm to copy your database over to it when you want to query it.
you would query against the live database, intorducing a potential DOS attack vector into your forum software?
well be that as it may, I won't
postgres, MsSQL, MySQL MariaDB, Redis, MongoDB, i don't care what the engine is, i'm not enabling another DOS attack vector. particularly one that could be triggered by a single user.
-
@FrostCat said in Reputation vs. Posts:
@anotherusername said in Reputation vs. Posts:
a leading 0
No, I get that. I remember the ANSI edition of K&R, and everybody's favorite trivial change. I was boggling at the idea that that block of text should've been interpreting stuff as octal.
hljs say: ruby. So ruby it is. Definitely. What else could it possibly be?
-
@FrostCat said in Reputation vs. Posts:
How big is the database again?
This is what the admin panel reports:
Data Size 7773.99 mb
Storage Size 9869.87 mb
Index Size 6470.07 mb
File Size 20407.00 mb
-
@boomzilla said in Reputation vs. Posts:
Where were all you people when we opened up the Migration category?
oh, where they always were, waiting for the decision to be made, and the migration to happen so they could complain about how it was done wrong.
-
@accalia said in Reputation vs. Posts:
absolutely.
that's why the only way i'm writing another shadowmod is if it has offline db access refreshed regularly. that way you can DOS shadowmod (by accident or malice) but you can't DOS the forum through that vector.Just rate limit requests or have a queue.
-
@boomzilla said in Reputation vs. Posts:
@Lorne-Kates said in Reputation vs. Posts:
Why are we using it?
It was the least worse thing we could find, apparently. Were you sleeping?
Probably. Not anymore, though.
-
@accalia said in Reputation vs. Posts:
you would query against the live database, intorducing a potential DOS attack vector into your forum software?
Pretty much everything else on the live site queries the live database, so why not? The real question is how heavy the queries are.
-
@loopback0 said in Reputation vs. Posts:
@accalia said in Reputation vs. Posts:
absolutely.
that's why the only way i'm writing another shadowmod is if it has offline db access refreshed regularly. that way you can DOS shadowmod (by accident or malice) but you can't DOS the forum through that vector.Just rate limit requests or have a queue.
and what about the monster query that never ends?
a rate limit or a queue won't save me from a query that never terminates until the database crashes.
i'm not risking it. i know what you monsters are like with the forums on a good day, i'm not letting what i write cause a bad day if i can possibly avoid it. not again.
-
@boomzilla said in Reputation vs. Posts:
@FrostCat said in Reputation vs. Posts:
Should I spell out that any of those would've been a better DB than Mongo?
Would that make you happy?
Where were all you people when we opened up the Migration category?
Probably assuming that the new forum would use a db. It's a pretty safe assumption, usually.
-
@boomzilla said in Reputation vs. Posts:
Where were all you people when we opened up the Migration category?
I'm not gonna tilt at windmills about it like @blakeyrat. I'm just participating in this convo. I don't know how much effort it'd take to get NodeBB to work atop a real RDBMS.
Today I'm just pointing out the idea that you can't do queries against the database in real time is an artifact of the shitty[1] "database" we're actually using.
[1] in this context.
-
@anotherusername said in Reputation vs. Posts:
What else could it possibly be?
Plain text. I don't know if there's a "shell" style.
-
@FrostCat said in Reputation vs. Posts:
@anotherusername said in Reputation vs. Posts:
What else could it possibly be?
Plain text. I don't know if there's a "shell" style.
database.txt
-
$ du -hs mongo-backup-2016-05-09/ 5.5G mongo-backup-2016-05-09/
-
@FrostCat said in Reputation vs. Posts:
The idea that you just can't do real-time queries on something in the tens of gigs...well, I can't quite come up with the right insult.
Considering that lots of real time queries is essentially what the forum is doing when people are using it...
-
-
@Lorne-Kates said in Reputation vs. Posts:
Probably assuming that the new forum would use a db. It's a pretty safe assumption, usually.
There was plenty of discussion about this. I think the default was to run on Redis, which I don't get at all, because supposedly there's no way to persist stuff to disk. I still don't think that can possibly be right, but who knows?
Then we switched to mongodb, which also seems crazy to me, who loves him some SQL. But I don't miss the SQL of discourse at all (well, maybe the statsporn a little).
-
-
@FrostCat said in Reputation vs. Posts:
I don't know how much effort it'd take to get NodeBB to work atop a real RDBMS.
@Weng started some work on that, but I assume the memory of it has been washed away by his slave drivers at WtfCorp at this point.
-
@FrostCat well, the language type hint can be used to tell it that, but nothing was specified and for whatever reason it buttumed that it was Ruby.
-
@FrostCat nope.
$ du -hs /usr/share/nginx/wtdwtf-nodebb.uploads/ 11G /usr/share/nginx/wtdwtf-nodebb.uploads/
-
@accalia said in Reputation vs. Posts:
and what about the monster query that never ends?
They're pre-written queries - I'm not sure how a never ending query can be introduced?
-
@anotherusername said in Reputation vs. Posts:
Pretty much everything else on the live site queries the live database, so why not? The real question is how heavy the queries are.
Exactly. I mean, you would be insane to run unvetted queries. If you get into the hundreds of gigs, then yeah, even good queries against stupidly-large data sets will run poorly...but this just cements my feeling that shouldn't do database design.
-
@boomzilla said in Reputation vs. Posts:
because supposedly there's no way to persist stuff to disk. I still don't think that can possibly be right, but who knows?
It can persist to disk, but the entire DB has to be in memory at all times.
-
@accalia said in Reputation vs. Posts:
and what about the monster query that never ends?
You don't have a backup of the database to test queries against?
-
@loopback0 said in Reputation vs. Posts:
@FrostCat said in Reputation vs. Posts:
The idea that you just can't do real-time queries on something in the tens of gigs...well, I can't quite come up with the right insult.
Considering that lots of real time queries is essentially what the forum is doing when people are using it...
The type of query and the hardware involved make massive differences. Remember when the web spiders were going nuts indexing all the old categories where no one was otherwise looking?
Our setup seems pretty good for our normal load, but adding analytics queries (for which mongodb seems to be legendarily bad at) is a terrible idea.
-
@FrostCat said in Reputation vs. Posts:
@accalia said in Reputation vs. Posts:
and what about the monster query that never ends?
You don't have a backup of the database to test queries against?
you can test the query against every possible input of parameters and database data configuration?
you might be willing to risk querying the live DB for play purposes..... bit i'm not.
-
@loopback0 said in Reputation vs. Posts:
Considering that lots of real time queries is essentially what the forum is doing when people are using it...
Yes. I mean, when I was working at Lucent, I had a report that had a ten minute query, but it was doing hundreds of thousands of nested loops against a DB in the hundreds of gigs, doing BOM explosions on every future order. Normal queries were much faster, even with huge amounts of data.
-
@boomzilla said in Reputation vs. Posts:
But I don't miss the SQL of discourse at all
Well, but that's because it had shit indexing, wasn't it? And, from what little I saw, a retarded schema, too.
-
@boomzilla said in Reputation vs. Posts:
but adding analytics queries (for which mongodb seems to be legendarily bad at) is a terrible idea.
I'm sure. I'm just saying, given a good schema and underlying RDBMS, I bet it wouldn't be.
-
Can MongDB easily be exported to something that Postgres or MySQL could consume?
-
Disclaimer: the sudden drop at the end of the last chart is because it is currently only 13:06.
-
@accalia said in Reputation vs. Posts:
you can test the query against every possible input of parameters and database data configuration?
you might be willing to risk querying the live DB for play purposes..... bit i'm not.Uh, that's my day job. I run queries against live DBs literally almost daily, undoing screwups my clients make. But I know how indexes work, how to design good ones, and how to use them effectively in queries.
-
@loopback0 said in Reputation vs. Posts:
Can MongDB easily be exported to something that Postgres or MySQL could consume?
yes, it can be exported, just run @ben_lubar's import process backwards.
easily?
/shrug i don't know his process.
-
@accalia hey, if you want to write a NodeBB-to-Discourse importer, I won't stop you.
Or go anywhere near it.
-
@FrostCat said in Reputation vs. Posts:
Uh, that's my day job. I run queries against live DBs literally almost daily, undoing screwups my clients make. But I know how indexes work, how to design good ones, and how to use them effectively in queries.
you also, presumably, have some manner of control over the database schema and indexes used? I don't, not for sockbot.
-
@accalia said in Reputation vs. Posts:
you also, presumably, have some manner of control over the database schema and indexes used?
Obviously.
I don't, not for sockbot.
Fair enough. I wouldn't let anyone touch my DB with random queries that haven't been tested, either. But stuff that's been vetted--and certainly someone like @ben_lubar could test something like sockbot against a copy of the DB, at least in theory.
-
@FrostCat said in Reputation vs. Posts:
I mean, you would be insane to run unvetted queries.
I'm not talking about hundreds. I'm talking about whatever the Mongoloid version is of:
SELECT u.user, u.user_name, p.post_count, r.reputation, CASE WHEN reputation = 0 THEN 0 ELSE post_count / reputation END as ratio FROM users as u INNER JOIN ( SELECT user, count(1) as post_count FROM posts WHERE topic_id <> @likes_topic ) as p ON p.user = u.user INNER JOIN ( SELECT user, sum(reputation_mod) as reputation FROM reputation_history WHERE topic_id <> @likes_topic) as r ON r.user = u.user
Hell, post count and reputation may already be pre-calced somewhere, making it even easier.
This isn't Deep Though level shit.