:fa_bullhorn: The sound of AN ANNOUNCEMENT BEING MADE (or: Request for Comments: Comments)
-
If I'm reading the EXPLAIN output correctly, it's already doing that. In any case, the import of ~800k posts took about 7 hours.
-
RE: invalid topics titles
sometimes I just give up, comment this line out https://github.com/NodeBB/NodeBB/blob/v0.9.2/src/topics/create.js#L30 and go on with my life - it imports them just fine, its just these 2 topics won't have a slug, which is fine because NodeBB uses the tid for the routing, not the full path= id + / slugRE: I'm seeing a lot comments in thread about HTML markup not being cleaned up, but since I can't follow the flow of this conversation, I'm gonna say this:
- after the import is done, you can use Post-Import Tools > Convert > HTML To Markdown, or BBCode to Markdown, the latter is better, and it converts html along the way, screenshot: https://cloud.githubusercontent.com/assets/1398375/11704596/f66bf7c8-9eb6-11e5-89c6-03268dc3b4ed.png,
it's using https://github.com/akhoury/bbcode-to-markdown which means if you find new tags that are not being converted, I can add them - you can also pass in your own javascript logic if you have some fancy regexp operations you want to do, either before the main convert or after. You can do the conversion thing as many times as you want, the importer keeps an original copy of each record's content, title, user's sig etc .... (that's why your new nodebb database is probably double in size) and it uses the original every time - don't worry, after you're all done and happy with the import process you can delete the extra data to save space, but you won't be able to use tools.
one last thing, please file issues on github, i dont usually come here.
-
after you're all done and happy with the import process you can delete the extra data to save space
Will that delete the sorted sets that map exported IDs to imported IDs? Because we need those for redirects to work.
-
Hmm, importing Community Server quotes will be hard, since the usernames don't match.
-
yes - just don't delete anything till you're 100% sure you're done, and of course after you've already generated and dowloaded a copy of that Map
Or simply dont ever delete it, and deal with the extra space, what is it, less than 1GB in your case?
-
I've got all of the users, topics, and posts imported and about a quarter of the votes, and the 16GB postgres database and 4GB mssql database combined to make a 12GB mongo database. Magic combined with anti-matter, I guess.
The problem with generating a map is Discourse's really stupid URL style (look at this topic's URL - who puts an ID after the slug?). I'm thinking maybe I should make a copy of the ID mappings we need during Exporter.teardown.
It looks like we need to keep
_imported:_users
,_imported:_posts
,_imported:_topics
, and_imported:_categories
. But those are just mappings between two sets of integers, so probably not a problem.
-
You know Microsoft SQL Server draws little boxes and arrows so you can actually read shit like that.
-
And then you need 3rd party tools like SQL Sentry to even zoom out of the resulting mess of boxes, arrows and lines.
-
I don't. But I guess that's just because I'm so much better than you.
-
No, you just write only really simple queries.
-
If you'd like, you could generate a diagram for this much simpler query that returns about a quarter of the number of results.
-
No, you just write only really simple queries.
That's a synonym for "I'm better than you".
If you'd like, you could generate a diagram for this much simpler query that returns about a quarter of the number of results.
Pfft. What are you paying me? Cash-moneys up front.
-
i see ... ok then, we can write a custom script to delete the extra stuff minus the stuff that you're using for redirects.
-
That's a synonym for "I'm better than you".
Uh-huh.Seriously though, try it out some time:
http://www.sqlsentry.com/products/plan-explorer/sql-server-query-viewIt does other nice things than just zooming. And it integrates into SSMS. It was even recommended to me by Microsoft employees during a SQL Server performance training. And yes, they admitted that SSMS (2012-2014, haven't tried 2016 yet) sucked in that regard.
-
What the hell. On clicking that link I get a new blank tab, the link never loads. Discourse..
-
Works for me.
-
Hmm, importing Community Server quotes will be hard, since the usernames don't match.
I've had to deal with shit like that before. I just end up with some sort of map between usernames, so you know:
- Community Server Username / ID
- Dickyolk Username / ID
Of course, if we hadn't changed people's usernames between CS and DickCock... like adding an underscore...
-
I have a map between CS ID and DiscoID. Community Server doesn't use textual URLs anywhere. Plus there are a lot of quotes on CS that aren't from a real user because people make jokes with those. Probably best to keep quotes not tied to users.
-
I have a map between CS ID and DiscoID. Community Server doesn't use textual URLs anywhere. Plus there are a lot of quotes on CS that aren't from a real user because people make jokes with those. Probably best to keep quotes not tied to users.
As long as they render as quotes, and have a username in them, I think it'll be fine. If someone wants to go, post-import, and do some sort of data-update trickery with those old quotes so they become whatever'd, then yay.
-
Go to /admin/plugins/sanitizehtml and enter the following in the fields:
Allowed tags:
[ "h1", "h2", "h3", "h4", "h5", "h6", "blockquote", "p", "a", "ul", "ol", "nl", "li", "b", "img", "i", "strong", "em", "strike", "code", "hr", "br", "div", "table", "thead", "caption", "tbody", "tr", "th", "td", "pre", "ins", "del", "abbr" ]
Allowed attributes:{"a": [ "href", "name", "target" ], "img": ["src", "class", "alt", "title"], "abbr": "title" }
Self closing:[ "img", "br", "hr", "area", "base","basefont", "input", "link", "meta" ]
Can anyone think of any tags I missed?
-
Details and summary.
-
-
What's a "reputation"? Is there something similar to "likes" that managed to be imported as well?
-
It's like likes but you can also downvote.
-
Ew, roundvatars!
-
We can fix that quite easily because NodeBB has support for custom CSS.
Filed under: I thought about disabling the rounding for the screenshot but that would be cheating
-
As this is postgresql...
You can use pgAdminIII for graphical EXPLAINS. Here's a nice intro...
-
Why I didn't @ben_lubar that...
Anywho! Also look at pgAgent:
pgAgent is a job scheduler for PostgreSQL, similar to Microsoft's SQL Server Agent. It's designed to run on a variety of operating systems, … and allows authorised users to schedule jobs consisting of one or more SQL or shell/batch steps. Common uses may include scheduled maintenance tasks - for example, archival of old records from a table, or vacuuming of certain tables, or operations that for part of an application such as batch posting of the days transactions to accounts…
-
Good job, @ben_lubar!
One small suggestion (that is absolutely Low prio): make the smaller. I don't think it should be much bigger than the icons next to it, or it'll look out of place.
-
Can anyone think of any tags I missed?
tt, code, kbd, dl, dt, dd, tfoot, summary, details, video, and if you're also referring to cooked, aside.Also, hope you have a class whitelist or else welcome back fa-spin.
-
I didn't notice this earlier, but...
-
tt
That's deprecated and also it's much easier to type
`foo`
than<tt>foo</tt>
, but whatever. Added.code
Already in there.
kbd
,dl
,dt
,dd
Added. Not sure if the CSS is set up.
tfoot
,summary
,details
Added.
video
What attributes should I allow? Also, what attributes on
audio
?and if you're also referring to cooked,
aside
.NodeBB uses
<blockquote>
.Also, hope you have a class whitelist or else welcome back
fa-spin
.Shit, why did the default stuff have class as an allowed attribute?
-
https://what.thedailywtf.com/t/lounge-title-redacted/47495/1107
Congratulations on finding a Signature Guy exploit in NodeBB, lounge user who I will not name!
-
Apparently this bug has been known for months.
-
-
Guess whose HTML sanitizer doesn't have that bug?
Should I write a NodeBB plugin that uses that library I wrote three years ago?
-
You're cheating by TESTING. Where's the fun in that?
-
Looked to me like it was a div rather than an a tag.
Also, this is surely not the only incident of unclosed tags across the forum, it's long been a way to hide crap in cooked posts.
-
Yeah, I have a lot of
<a
s in my posts.......
-
-
https://what.thedailywtf.com/t/lounge-title-redacted/47495/1107
Congratulations on finding a Signature Guy exploit in NodeBB, lounge user who I will not name!
ara? how did that work?!
i see nothing special about the linked post......
Issue: Unclosed
<a>
tag breaks tag balance..... oh....... Yeah there's gonna be a lot of those for me. i throw those things in when i get
500 OK
followed by "Body is too similar"so yay me?
-
Oh, it's because the sanitizehtml plugin uses 0.1.4 instead of 1.11.3.
wat
-
Ok, this is weird. That fixed
<div
, but not<div foo
.Before sanitization: <p>Hey! Here is a broken link tag: <html title After sanitization: <p>Hey! Here is a broken link tag: </html>
-
HTML sanitisation is hard.
-
Yeah, I have a lot of
<a
s in my posts.......My plan:
- Get a prude installed as a sockpuppet mod
- Secretly install code that executes on <butt tags
- post hyperlinks for <a ss
-
HTML sanitisation is hard.
s/</</g
not so hard.
oh, you wanted to whitelist some tags?
that's not sanitzation anymore! :-D
-
-
Ok, here's the list of things I still need to do:
- private messages
- polls
- custom user profile fields
Let's start with private messages. To the vim room!
-
This topic is already quite old and I enjoy reading it.
I am not sure if anybody else has said thisalready (if so: ) but:
#Thank You for working so hard on the transition between the forums.
Filed Under: there, I even used #
-
there, I even used #
#
works like people originally expected it to on NodeBB.<h1>
is the replacement.